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ABSTRACT 


This  report  describes  s  possible  spprosch  for  providing  more  accu¬ 
rate  probability  assessments  In  those  real-world  decision-making  situa¬ 
tions  for  which  relative  frequency  Information  generally  does  not  exist* 
In  this  approach  an  estimated  functional  relationship  Is  obtained  from  a 
comparison.  In  relative  frequency  base  (RFB)  problems,  of  a  person's 
posterior  probability  assessments  with  the  true  posterior  probabilities. 
Then,  the  functional  relationship  Is  used  as  a  model  to  provide  prob¬ 
ability  predictions  for  new  problems  by  adjusting  the  person's  corre¬ 
sponding  probability  assessments.  Experimental  evidence  Indicates  that 
the  resulting  models  can  provide  satisfactory  probability  predictions 
for  problems  having  Identical  structure  to  those  used  In  model  construc¬ 
tion.  At  this  time,  some  obstscles  prevent  Immediate  use  of  these 
models  for  probability  prediction  In  non-relative  frequency  base  (NRFB) 
problems.  However,  It  would  be  unwise  to  discount  the  feasibility  of 
developing  a  practical  RFB-based  model  for  adjusting  real-world  prob¬ 
ability  estimates  until  the  approach  were  tested  with  participants  and 
problems  In  a  field  of  common  expertise  (such  as  Intelligence  analysis 
or  weather  forecasting,  for  example)  where  more  uniformity  of  background 
and  experience  exists. 
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1 .  INTRODUCTION 


Decision-nicking  Is  usually  a  complex  and  difficult  task.  In  ixany 
cases,  however,  this  task  may  be  greatly  simplified  by  the  application 
of  numerical  techniques,  particularly  If  probabilities  are  used  to 
express  relationships  and  uncertainties  within  the  structure  of  the 
underlying  problem.  Thus,  probability  estimation  (or  probability  assess¬ 
ment^)  Is  a  valuable  step  In  the  declslon-naklng  process. 

The  Intelligence  analyst,  for  example.  Is  often  called  upon  to 
Implicitly  or  explicitly  estimate  probabilities.  Traditionally,  he  has 
reflected  his  Judgments  about  "raw"  Intelligence  according  to  source 
reliability  and  content  vail  ty  using  a  letter  scale  of  A  through  F 
for  source  reliability  and  a  numerical  scale  of  1  through  6  for  content 
validity.  In  addition,  or  as  a  substitute,  the  analyst  has  expressed  his 
Judgments  In  verbal  descriptions  such  as  "it  Is  doubtful  that",  "chances 
are  good  that",  "it  is  fairly  certain  that",  etc.  Unfortunately,  verbal 
descriptions  of  the  uncertain  link  between  data  and  predictions  may  tend 
to  confuse  rather  than  clarify.  This  contention  was  borne  out  in  a  study 
by  Johnson  (1973),  who  found  experimental  Indications  that  "...the  use  of 
qualitative  expressions  to  communicate  the  accuracy  or  relative  likelihood 
of  occurrences  of  Intelligence  data  and  products  will  often  result  In  a 
high  degree  of  misunderstanding." 

- 

Brown,  Kahr,  and  Peterson  (1974)  use  the  term  "probability  assess¬ 
ment"  when  the  true  probability  Is  unknown,  reserving  the  term  "probability 
estimation"  for  those  situations  In  which  the  true  probability  may  be  deter¬ 
mined.  This  distinction  Is  not  being  made  In  this  report;  "estimation"  and 
"assessment"  are  used  Interchangeably . 
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Numerical  probability  statements  (e.g.,  "I  believe  there  Is  a  70Z 
chance  that  event  E  will  occur.")  may  well  provide  a  more  understandable 
means  of  connunicat ing.  As  noted  by  Brown  aod  Shuford  (1973),  the  use 
of  numerical  probabilities  to  express  uncertainty  provides  concise, 
relevant  quantitative  data.  In  general,  however,  it  is  extremely  diffi¬ 
cult  to  asseas  the  calibration  (l.e.,  the  accuracy)  of  numerical  probability 
estimates  unless  relative  frequency  information  exists.  Unfortunately, 
such  Information  is  not  available  in  most  real-world  decision-making  situ¬ 
ations. 

The  fundamental  tenet  of  the  research  described  in  this  report  is 
that  the  performance  of  persons  making  probability  estimates  in  problems 
having  a  relative  frequency  base  (REB)  can  yield  information  with  which  to 
calibrate  their  estimates  made  in  situations  lacking  a  relative  frequency 
base.  The  reason  for  using  REB  situations  is  that,  unlike  non-relative 
frequency  base  (NRFB)  problems,  probability  estimates  made  in  REB  problems 
can  be  compared  with  a  normative  Bayesian  model  to  derive  models  of  each 
Individual's  inherent  estimation  biases  and  errors.  These  functional 
modelscan  then  be  used  to  adjust  the  estimates  made  In  NREB  probability 


estimation  tasks. 


I I .  BACKGROUND 


In  the  decision-making  framework,  expression  of  uncertainty  by 
means  of  numerical  probabilities  Involves  not  only  probability  estima¬ 
tion,  but  also  the  attendant  task  of  updating  probabilities  In  light 
of  new  information.  Most  experiments  to  Investigate  probability 
estimation  and  updating  have  Involved  "book-bag  and  poker  chip" 
problems.  The  following  description  of  these  problems  Is  given  by 
Winkler  and  Murphy  (1973). 


A  comonly  used  device  In  probsblllty  revision  experi¬ 
ments  Is  the  bookbag-and-poker-chip  paradigm,  In  which  the 
experimenter  has  a  number  (usually  two)  of  bookbags  filled 
with  poker  chips.  The  composition  of  the  bookbags  differs 
with  respect  to  the  number  of  poker  chips  of  a  particular 
color;  the  first  bookbag  may  contain  70  red  chips  and  30 
blue  chips,  for  example,  whereas  the  second  bookbag  contains 
30  red  chips  and  70  blue  chips.  The  subject  is  told  that 
one  of  the  bookbags  has  been  chosen  by  a  random  device, 
and  his  task  Is  to  assess  probabilities  for  the  possible 
bookbags.  Since  the  subject  is  told  that  the  bookbag  has 
been  selected  randomly,  the  Initial,  or  prior,  probabilities 
should  be  equal.  The  subject  Is  then  given  Information  In 
the  form  of  a  series  of  draws,  at  random  and  with  replace¬ 
ment,  from  the  chosen  bookbag.  Upon  observing  the  results 
of  these  draws,  the  subject  assesses  his  posterior  prob¬ 
abilities  for  the  possible  bookbags. 


Thus,  these  problAa  have  a  relative  frequency  base  (RFB)  as  do  the 
"balls  and  urns"  problems  often  used  In  Introductory  probability  courses 
to  Illustrate  the  application  of  Bayes'  theorem. 


A.  BAYES'  THEOREM  AND  DISCORDANCE 


Bayes’  theorem  serves  as  an  optimal  normative  model  for  updating 


probability  estimates.  The  uaual  form  of  this  theorem  is 


P(H  ]D) 

I  1 


P(D|H1)P(H1) 

rp(DiHt )P(H  ) 
k  k  k 


where  P(U^)  denotes  the  prior  probability  of  hypothesis  H^,  P(D  H^) 
denotes  the  conditional  probability  of  specific  data  D  occurring  given 
that  hypothesis  is  true,  and  P(H J  D)  denotes  the  posterior  probability 
of  hypothesis  H  given  the  occurrence  of  the  data  D. 

Numerous  studies*  have  indicated  what  has  been  called  "conservatism" 
in  estimation  of  P(H^|D),  posterior  probability.  This  terminology  was 
used  because  most  persons  tend  to  revise  their  estimates  from  the  prior 
probability  by  an  amount  which  is  less  than  that  indicated  by  Bayes' 
theorem.  The  more  general  term  "discordance"  will  be  used  in  this  report 
to  reflect  any  difference  between  an  estimated  probability  and  the  true 
probabl lity . 

Much  research  has  been  aimed  at  determining  hew  departures  from  the 
normative  Bayesian  model  are  affected  by  various  aspects  of  the  experi¬ 
mental  situation,  and  at  developing  theories  to  account  for  discordant 
behavior.  In  general,  the  emphasis  has  been  on  conservatism,  which  has 
been  found  to  depend  on  a  number  of  factors,  such  as  the  response  mode, 
the  dlagnost icity  of  the  data,  the  perceived  prior  probabilities,  the 
length  of  the  data  sequence  presented,  and  the  coaiposltlon  of  the  data. 

In  RFB  problems  estimates  of  P(D  H^)  are  usually  accurately  and 

I 

See,  for  example,  Edwards  (1966b),  Phillips,  Hays,  and  Edwards 
(1966),  Wheeler  and  Beach  (1968),  Chlnnls  and  Peterson  (1970),  and  Goodman 

(1972). 


easily  made.  Therefore,  the  true  value  of  P(H^  D)  can  be  obtained  without 

difficulty  by  application  of  Bayes'  theorem.  However,  In  the  real  world 

direct  estimates  of  P(H  1  D)  cannot  be  based  on  a  direct  relative  frequency 

1 

assessment.  Hence,  estimation  of  posterior  probabilities  Is  a  more  complex 
task.  It  should  be  stressed  that  In  most  real-world  (e.g.,  Intelligence) 
situations  neither  P(H^j  D)  nor  P(D|h^)  has  a  relative  frequency  basis. 

Thus,  both  these  types  of  probabilities  are  difficult  to  estimate,  and 
it  is  quite  likely  that  both  are  inaccurately  estimated.  If  this,  in 
fact.  Is  the  true  situation,  estimated  posterior  probabilities  would 
usually  be  Incorrect  regardless  of  whether  they  were  obtained  directly  by 
estimates  or  indirectly  by  Bayes'  theorem. 

Therefore,  discordance  Is  to  be  expected  in  estimates  of  P(D|Hj). 

Then,  despite  the  use  of  any  computer  aids,  the  value  of  P(H^|D)  arrived 
at  by  using  Inaccurate  estimates  of  P(D  H^)  In  conjunction  with  Bayes' 
theorem  may  be  no  more  accurate  than  a  direct  estimate  of  that  posterior 
probability.  Thus,  methods  of  evaluating  and  correcting  for  discordance 
in  probability  estimates  are  required  In  order  to  improve  the  quality  of 
those  estimates. 

B.  RESEARCH  OVERVIEW 

The  research  described  In  this  report  is  aimed  at  development  of  an 
algorithm  for  predicting  and  correcting  discordance  In  those  real-world 
decision-making  situations  for  which  relative  frequency  Information  link¬ 
ing  data  with  hypotheses  generally  will  not  exist.  This  research  has 
concentrated  on  the  simplest  type  of  problem,  the  tvo-hypothesls  problem 
with  equal  prior  probabilities. 


-5- 


The  rationale  underlying  the  decision  to  deal  with  this  type  of 
problem  rests  on  the  assumption  that  if  the  suggested  algorithm  does 
not  prove  promising  when  based  on  uncomplicated  examples,  there  would 
be  little  hope  that  it  would  prove  promising  on  more  complex  (and 
realistic)  problems  of  the  type  found  in  Army  tactical  decision-making 
Furthermore,  by  restricting  attention  to  simple  problems,  the  participating 
subjects  would  be  able  to  complete  more  problems  within  a  given  period 
of  time.  Hence,  more  data  would  be  available  for  estimating  the  required 
functions  and  for  testing  the  validity  of  the  suggested  algorithm.  Thus, 
at  this  stage  of  the  research,  the  use  of  a  number  of  short  straightforward 
problems  would  appear  to  offer  a  higher  payoff  than  the  use  of  a  smaller 
number  of  more  complex  (and  perhaps  more  realistic)  problems. 

Unlike  the  "book-bag  and  poker  chip"  problems,  those  problems  lacking 
a  relative  frequency  base  offer  no  direct  method  for  assessing  the  differ¬ 
ences  between  estimated  and  true  probabilities,  either  for  P(H^  D)  or  for 
P(DjH^).  Because  of  this,  the  general  procedure  must  be  an  Indirect  one. 

In  designing  an  experiment  for  observing  behavior  in  RFB  problems, 
consideration  must  be  given  to  its  fundamental  purpose.  If  the  objective 
were  to  obtain  estimates  which  match  as  closely  as  possible  those  from 
the  normative  Bayesian  model,  then  the  experiment  should  employ  the  tech¬ 
niques  which  prior  research  has  shown  to  reduce  discordance.  However,  the 
goal  here  is  to  observe  probability  eatlmatlon  in  RFB  situations  in  order 
to  develop  models  which  can  be  used  In  calibrating  probability  estimates 
made  in  NRFB  situations.  As  a  consequence,  the  present  experiment  sllowed 
the  responses  elicited  for  RFB  problems  to  exhibit  the  individual  biases 
and  errors  inherent  in  each  person's  approach  to  probability  estimation. 
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For  a  sore  rigorous  tiatement  of  the  hypothesis  underlying  the 
research,  suppose  that  I  people  are  Involved  In  a  probability  estimation 
task.  It  Is  reasonable  to  assume  an  underlying  functional  relationship  of 
the  generic  form 

P  -  f^r)  +  e 

where: 

p  denotes  the  true  probability  value 
r  denotes  an  estimated  probability  value 

fj  denotes  the  functional  relationship  between  p  and  r  for 
person  1 

e  denotes  an  error  term  resulting  from  random  variation 
present  In  a  person's  probability  estimates. 

It  should  be  noted  that  although  probability  values  are  assumed  here  for 
Illustrative  purposes,  one  or  both  of  the  variables  p  and  r  could  be 
some  tranaf ora  of  probability.  The  functional  relationship  fj  Is 
somewhat  akin  to  the  "realism  line"  discussed  by  Sibley  (1974).  The 
realism  line,  which  may  be  constructed  from  a  person's  probability  esti¬ 
mates  of  confirmable  future  events,  is  a  plot  of  observed  relative 
frequency  vs.  estimated  probability. 

The  realism  line  concept  is  related  to  the  Idea  of  a  scoring  rule. 

As  described  by  Brown,  Kahr  and  Peterson  (1974),  a  scoring  rule  Is  a 
device  for  providing  feedback  to  an  estimator  so  that  he  may  be  rewarded 
In  terms  of  how  well  his  forecasts  turn  out.  A  proper  scoring  rule  Is 

one  by  which  the  assessor  can  maximize  his  score  only  by  reporting  his 

* 

actual  beliefs,  since  any  other  estimation  or  reporting  strategy  will 
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produce  a  lower  score.  A  realism  line  derived  for  a  subject  can  be 
evaluated  in  terms  of  a  proper  scoring  rule.  Conversely,  application 
of  a  proper  scoring  rule  can  be  used  to  generate  a  realism  line  from  a 
set  of  observations. 

Unfortunately  the  use  of  scoring  rules  and  realism  lines  as  decision 
aids  has  several  practical  disadvantages,  the  foremost  of  which  is  that 
one  must  obtain  assessments  for  a  large  number  of  confirmable  events 
and  await  their  confirmation  before  the  assessor  can  be  Informed  of  the 
results.  This  further  implies  that  the  assessor  must  be  available  first 
to  make  the  assessments  and  also  later,  after  the  events  have  been  con¬ 
firmed,  to  receive  the  feedback.  Another  disadvantage  is  that  any  form 
of  feedback  ->r  use  of  scoring  rules  may  interfere  with  the  assessment 
process  and  thus  mav  inhibit  performance  rather  than  enhance  it.  The 
approach  adopted  in  the  current  research  obviates  these  difficulties  by 
avoiding  the  use  of  confirmable  events  where  the  ultimate  probabilities 
are  either  zero  or  one,  substituting  Instead  the  use  of  problem  situations 
involving  probabilities  covering  the  full  (0,1)  range. 

In  the  current  study,  an  estimated  functional  relationship  f  (not 
necessarily  of  the  same  form  for  each  person)  was  obtained,  via  regression 
techniques,  from  a  comparison  of  each  person's  posterior  probability 
assessments  in  RFR  problems  with  the  true  posterior  probabilities.  Then, 

A 

using  the  posterior  probability  estimate  r  made  by  person  1,  f^  should 
provide  a  prediction  (i.c.,  a  corrected  estimate)  p  of  the  true  posterior 

A  A 

probability.  In  mathematical  notation,  p  •  f  ^  ( r )  . 

By  looking  at  these  functions  for  several  people,  it  is  possible  to 
test  whether  or  not  the  discordance  observed  in  the  estimates  of  P(h|d) 
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In  a  non-relative  frequency  base  (NRKB)  problem  may  be  corrected.  Civen 
K  NRKB  problems  In  which  posterior  probabilities  are  to  be  estimated,  a 
probability  assessment  r  ^  may  be  obtained  for  each  person  and  problem. 

If  the  degree  of  discordance  Is  the  same  as  In  the  RFB  problems,  the 
predicted  "true"  probabilities  p..  for  1  •  1,...,I  [where  p  “  f  (r  )J 

IK  1  IK 

should  be  unbiased  estimates  of  the  same  quantity.  That  Is,  the  estimated 
functional  relationships  f^  should  provide  a  means  by  which  assessments  of 
NRKB  probabilities  may  be  calibrated. 

The  statistical  procedures  for  testing  the  feasibility  of  correcting 
for  discordance  are  described  in  Appendix  A.  That  appendix  Includes  a 
discussion  of  functional  relationships,  hypothesis  testing,  and  statis¬ 
tical  power  considerations. 


-9- 


1 1 1 .  THE  EXPERIMENT 


If  the  degree  of  discordance  In  the  NRFB  problems  were  found  to 
be  the  same  as  in  the  RFB  problems,  significant  implications  would 
exist  for  potential  application  in  Army  tactical  decision-making.  For 
example,  the  functions  f^  could  be  used  directly  as  a  feedback  mechanism 
with  which  to  train  probability  estimators  to  Improve  their  accuracy. 

As  an  alternative,  the  functions  could  Instead  be  used  as  a  correction 
device  to  minlaixe  bias  in  probability  estimates.  This  latter  applica¬ 
tion,  It  should  be  noted,  would  not  Interfere  with  an  analyst's  method 
of  estimating  pi obab 1 1  it les  nor  would  it  require  that  probability  esti¬ 
mators  ecelve  special  training  to  help  them  change  their  probability 
assessments . 

To  test  the  feasibility  of  correcting  for  discordance,  an  experi¬ 
ment  was  performed  following  the  design  discussed  in  Appendix  A. 
Participants  in  the  experiment  were  Pennsylvania  State  University  students, 
who  made  probability  estimates  for  problem  situations  which  had  been 
specifically  designed  for  this  study.  The  main  experiment  was  preceded  by 
a  pretest  using  a  subset  oi  the  problem  situations,  for  which  estimates 
were  made  by  students  from  Catholic  University. 

A.  METHOD 

The  materials  used  in  this  experiment  included  a  set  of  three  types 
of  problem  situations  (RFB,  NRFB,  and  almanac).  The  problem  situations 
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were  preceded  by  Instructions  explaining  the  probability  estimation  task 
and  by  practice  Items.  A  questionnaire  ended  the  research  session.  The 
complete  set  of  materials  used  In  the  experiment  Is  reproduced  in  Appendix  B. 

1.  RFB  Problem  Situations 

The  RFB  Items  used  in  this  study  were  of  the  "book-bag  and  poker  chip" 
variety.  A  general  format  was  devised  and  rigorously  used  for  all  RFB  Items, 
with  the  specifics  being  systematically  changed  from  problem  to  problem. 

The  basic  RFB  format  may  be  seen  in  Figure  1,  which  exhibits  one  of  the  prob¬ 
lem  situations  used  In  the  experiment. 

The  subjects  were  told  in  each  RFB  problem  statement  to  assume  that 
there  were  two  large  Identical  bags  each  filled  with  a  large  number  of 
ordinary  poker  chips.  Some  of  these  chips  were  red  and  some  were  blue. 

Bag  "8"  contained  predominantly  blue  chips  In  a  certain  ratio,  while  bag 
"R"  contained  predominant ly  red  chips  In  the  same  ratio.  Thus,  If  bag  "B" 
contained  642  blue  chips  and  362  red  chips,  bag  "R"  contained  642  red  chips 
and  367  blue  chips. 

A  fictitious  coin  toss  for  selecting  a  bag  made  the  prior  probability 
of  selection  502  for  each  bag.  Sequential  one-chip  samples,  with  replace¬ 
ment  and  remixing  after  each  sample,  were  then  assumed  to  have  been  drawn 
from  the  chosen  bag.  The  results  of  the  samples  were  presented  to  the 
subjects  simultaneously.  The  subject's  task  was  to  estimate  the  (revised) 
probability  that  a  specified  bag  had  been  chosen,  based  on  the  Information 
given  In  the  sample  draws.  The  problem  statements  emphasized  that  the 
Initial  probability  was  502  and  that  replacement  and  remixing  occurred 
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Problem  Sit  use,  ion  #10 


Assume  that  there  are  two  large  Identical  baga,  "B"  and  "R”,  eacli 
filled  with  a  large  number  of  poker  chlpe.  Although  each  bag  containa 
the  aame  number  of  chips,  there  la  one  difference: 

Bag  "B”  containa  U5Z  blue  chipa  and  15Z  red  chipa 

Bag  "K"  containa  15Z  blue  chipa  and  B5Z  red  chipa 

Baaed  on  the  toaa  of  a  coin,  one  of  the  baga  la  aelected.  The  other 

bag  la  act  aalde.  At  this  point,  alnce  each  bag  ia  equally  likely  to  have 

been  aelected  by  the  coin  toaa,  it  la  reaaonable  to  eatlmate  that  the 
probability  la  SOZ  that  bag  "B"  was  choaen. 

Assume  that  a  person  who  la  aecurcly  blindfolded  drawa  chipa  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  aelected  bag  are 
thoroughly  mixed.  Then  one  chip  la  drawn.  The  chip  la  red.  The  chip  is 
returned  to  the  aelected  bag,  che  contents  are  again  mixed  and  a  second 
chip  la  drawn.  This  second  chip  is  blue.  The  chip  la  returned  to 
the  bag,  the  contents  are  rcmlxod  and  a  third  chip  la  drawn.  Thla  chip 
la  red.  The  chip  la  returned  to  the  bag,  the  contents  arc  remixed 
and  a  fourth  chip  la  drawn.  Thla  chip  la  blue.  The  chip  la  returned 
to  the  bag,  the  contents  arc  reoixed  and  a  fifth  chip  la  drawn. 

Thla  chip  ia  blue. 

Question:  What  would  you  eatlmate  to  be  the  probability  that  the 
selected  bag  la  bag  "B"? 


Suaaary 

Initial  probability  eatlmate:  50Z 
Information : 

(1)  First  chip  la  red 

(2)  Second  chip  la  blue 

(3)  Third  chip  la  red 

(4)  Fourth  chip  la  blue 
(3)  Fifth  chip  la  blue 


Answer : _ Z 


FIGURE  1:  An  RFB  Problem  Situation  Uaed  in  the  F.xperiment 
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after  each  draw. 


Following  the  full  narrative  description  given  in  each  problem 
situation,  including  the  bag  color  ratios  and  the  draw  sequence,  a  simpli¬ 
fied  summary  of  the  problem  conditions  waa  provided.  The  subject  then 
used  a  percentage  between  OX  and  100Z  to  estimate  the  probability  that  a 
specified  bag  had  been  chosen. 

There  were  no  instructions  given  for  interpreting  the  results  of  the 
sample  draws,  except  that  a  note  provided  after  the  RFB  practice  item 
explained  that  "since  more  blue  chips  were  drawn  from  the  bag,  it  is  reason¬ 
able  to  estimate  that  the  probability  that  bag  "B"  was  chosen  is  greater 
than  50Z  because  that  bag  contains  more  blue  chips  than  bag  "R"  does." 

For  the  class  of  RFB  problem  situations  used  in  this  study,  posterior 
probability  is  a  function  of  the  arithmetical  difference  between  the  number 
of  red  and  blue  chlpsf  It  was  therefore  found  efficient  in  constructing 
the  RFB  items  to  use  a  computer  program  to  generate  candidate  problem 
situations  as  a  function  of  bag  content  ratio  and  the  excess  of  one  color 
over  the  other  for  all  draws  within  the  sample.  Consideration  was  limited 
to  color  excesses  of  one,  two  or  three  chips  over  all  draws  within  a  prob¬ 
lem.  True  probabilit les  for  the  candidate  situations  were  arranged  in 
sequence,  and  twenty  items  were  selected  to  Include  two  true  probabilities 
within  each  decile.  Draw  sequences  were  established  so  that  each  sample 
contained  from  two  to  five  chips,  and  the  sequence  of  colors  of  the  draws 
was  randomized.  The  target  bag  was  then  designated  for  each  item,  and  the 
overall  sequence  of  the  RFB  problems  was  randomized. 

j 

See  Shanteau  (1972),  for  example. 
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Ten  of  the  RFB  problems  thus  constructed  were  used  in  the  pretest. 
Four  additional  problems  Included  in  the  main  experiment  were  randomly 
Inserted  in  the  sequence  of  problems  in  the  experiment.  The  four  added 
items  were  selected  to  have  true  probabilities  in  the  top  and  bottom  two 
deciles  of  the  range  in  order  to  provide  better  resolution  for  extreme 
probabilities.  The  basic  characteristics  of  the  fourteen  RFB  items  used 
in  the  main  experiment  are  sumarlzed  in  Figure  2.  The  pretest  sequence 
numbers  are  shown  in  the  right-hand  column. 

2.  NRFB  Problem  Situations 

The  NRFB  problems  used  in  the  experiment  were  general  information 
situations  constructed  so  as  to  have  a  format  similar  to  the  RFB  items. 

The  main  difference  between  the  RFB  and  NRFB  problems  was  that  the  latter 
lacked  a  relative  frequency  base,  and  thus  no  true  probability  could  be 
calculated.  In  addition,  whereas  all  the  data  elements  in  the  RFB  items 
were  the  same  (i.e.,  chip  draws),  a  variety  of  data  elements  were  used  in 
the  NRFB  Items.  An  example  of  a  NRFB  problem  is  given  in  Figure  3. 

The  nine  NRFB  problems,  which  were  used  in  both  the  pretest  and  main 
experiment.  Included  a  variety  of  topics.  The  varied  subject  matter  of 
these  problem  situations  mav  be  seen  in  Appendix  B. 

It  was  necessary  to  devise  NRFB  situations  which  fell  within  the 
range  of  experience  of  the  subjects  snd  which  were  relatively  easy  for  the 
average  person  to  grasp.  In  each  NRFB  problem  there  were  from  three  to 
five  data  elements.  To  the  greatest  extent  possible,  each  was  constructed 
to  be  conditionally  Independent  of  the  others.  These  data  elements  were 
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FIGURE  2  :  Character lstlcs  of  the  RFB  Problem  Situations 


Problem  Situation  #15 


Assume  chat  CM  sella  the  same  number  of  1978  trucks  that  are  sold 
by  non-GM  manufacturers.  Thus,  If  in  s  conversation  with  a  stranger, 
you  found  out  that  he  had  purchased  a  new  1978  truck,  It  would  be  reason¬ 
able  to  estimate  that  the  probability  is  50Z  that  his  truck  was  a  CM. 

Suppose  that,  at  the  beginning  of  the  next  model  year,  GH  sustains 
a  massive  strike  for  three  months  during  which  no  1979  trucks  are  produced, 
while  all  non-CM  truck  manufacturers  avoid  strikes.  Further  suppose  that, 
midway  through  the  model  year,  a  Ralph  Nader-type  group  Issues  a  report 
that  1979  CM  trucks  have  serious  safety  defects.  In  addition,  assume 
that  although  1979  GM  trucks  Increased  in  price  by  an  average  of  $190  over 
the  previous  year's  price,  non-GM  trucks  increased  in  price  by  an  average 
of  $345. 

Sometime  In  1980  you  meet  another  stranger  whom  you  find  has  purchased 
a  new  1979  truck. 

Question:  What  would  you  estimate  to  be  the  probability  that  it 
was  a  GM  truck? 


Suwsary 

Initial  probability  estimate:  50Z 
Information: 

(1)  GM  strike 

(2)  GM  safety  questioned 

^ ^  Smaller  GM  price  Increase 


Answer : 


Z 


FIGURE  3:  An  NRFB  Problem  Situation  Used  In  the  Experiment 
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designed  Co  Influence  a  respondent's  probability  estimate  In  a  specific 
direction  and  within  a  given  range  of  Intensity.  The  subjects  were,  for 
each  problem,  to  combine  the  overall  magnitude  and  direction  of  the  data 
elements  Into  a  single  weighted  assessment  of  the  posterior  probability. 

NXFB  items  can  quite  readily  be  constructed  to  elicit  a  fairly  con¬ 
sistent  response  by  generating  only  likely  (or  only  unlikely)  events 
conditional  on  a  specific  hypothesis  so  that  the  overall  effect,  by  almost 
anyone's  Inherent  weighting  scheme,  strongly  favors  one  hypothesis.  It 
Is  more  difficult  to  devise  a  set  of  general  Information  NRFB  problem 
situations  such  that  the  resultant  probability  estimates  span  a  range  of 
values.  One  obvious  way  to  obtain  an  estimated  probability  within  the 
middle  of  the  probability  range  Is  to  construct  a  set  of  data  elements  with 
a  consistent  trend,  but  with  low  to  moderate  Intensity  so  that  the  combination 
does  not  drive  the  response  to  the  extreme.  Another  way  Is  to  devise  a 
carefully  balanced  set  of  elements,  some  of  which  offset  one  another.  Of 
course,  subjects  may  differ  In  their  perceptions  of  Individual  data  elements, 
in  their  weighting  scheme,  or  In  both. 

In  an  attempt  to  obtain  coverage  of  the  probability  spectrum,  the 
experimenters  made  Independent  Judgments  of  the  general  position  of  the 
posterior  prolabllity  for  each  NRFB  problem.  Where  necessary,  adjustments 
were  made  to  problems  In  order  to  provide 

(1)  agreement  In  the  experimenters'  rough  assessments 
and  (2)  coverage  of  the  entire  (0,1)  probability  range. 

These  assessments  were  partially  confirmed  when  the  subjects'  average 
estimated  probabilities  were  calculated  for  the  pretest  and  the  main 
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experiment.  When  these  were  compared  with  the  exper lm 'nters '  assessments, 
there  was  agreement  for  seven  of  the  nine  N’RFB  Items.  Of  course,  It  was 
not  expected  that  there  would  be  complete  agreement  between  the  subjects 
and  the  experimenters. 

j.  Almanac  Problem  Situations 

A  third  type  of  problem  situation  employed  in  this  experiment  was 
based  on  information  obtained  from  the  1977  World  Almanac.  The  almanac 
Items,  which  could  not  be  put  Into  a  format  similar  to  the  RFB  and  NRFB 
problem  situations,  were  Included  In  the  experiment  In  order  to  gain 
Information  about  the  feasibility  of  uslnR  RFB-based  calibration  In  problem 
situations  which  did  not  have  a  structure  parallel  to  that  of  the  RFB 
problem*.  Figure  4  provides  an  example  of  an  almanac  problem  used  in  the 
experiment.  A  total  of  eight  almanac  problems  were  employed  in  both  the 
pretest  and  the  main  experiment.  These  problems  are  reproduced  In  Appendix 
B. 

Items  which  could  be  cast  In  probabilistic  terms  were  chosen  from 
the  almanac.  This  restricted  choice  to  situations  In  which  a  population  was 
broken  Into  constituent  elements,  su.  h  as  the  population  of  the  U.  S.  broken 
down  by  states.  An  additional  restriction  was  that  a  reasonable  sounding 
probability  question  had  to  be  possible  from  the  information.  It  seemed 
fairly  reasonable  to  ask  subjects  to  estimate  the  probability  that  a  baby 
born  In  the  U.  S.  during  1974  was  born  east  of  the  Mississippi,  since  the 
true  probability  could  be  established  in  terms  of  the  relative  occurrence 
of  births  by  states.  It  would  be  more  than  a  bit  strained,  however,  to 
ask  the  probability  that  a  square  mile  of  U.  S.  land  chosen  at  random  la 
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rob lea  Situation  #28 


FBI  figures  for  1975  report  that  there  were  116,000  arrests  In  the 
U.S.  that  year  for  robbery. 

Question:  What  would  you  estimate  to  be  the  probability  that  a  1975 
U.  S.  arrest  for  robbery  was  a  nan  rather  than  a  woman? 


Answer :  _ _ * 


FIGURE  4:  An  Almanac  Problem  Situation  Used  in  the  Experiment 


in  Alaska,  even  Chough  the  almanac  indicates  that  16Z  of  the  U.  S.  landmass 
is  in  that  state. 

B.  THE  PRETEST 

A  pretest  had  the  purpose  of  trying  out  and  evaluating  all  aspects  of 
the  test  administration  and  data  analysis  procedures.  However,  the  two 
most  Important  objectives  were  to  determine  how  much  time  should  be  allowed 
for  completion  of  the  problem  situations,  and  to  Judge  the  reasonableness 
of  the  RFB  and  NRFB  items  based  on  comments  solicited  from  the  participants. 

Subjects  for  the  pretest  were  students  in  an  introductory  psychology 
class  at  Catholic  University.  Arrangements  were  made  with  the  instructor 
to  administer  the  test  during  a  regular  class  period,  which  lasted  leBs 
than  an  hour.  Class  members  had  been  Informed  earlier  in  the  semester 
that  a  class  period  (date  unspecified)  would  be  devoted  to  an  experiment 
in  which  they  would  participate.  Twenty-two  persons  took  part,  but  since 
three  were  unable  to  complete  all  of  the  items  in  the  time  available,  only 
the  data  from  nineteen  subjects  were  included  in  the  analysis. 

Instructions,  including  a  brief  outline  of  the  probability  concepts 
involved  in  the  experiment,  were  read  aloud  to  the  pretest  subjects.  The 
experimental  materials  were  then  distributed,  and  the  subjects  proceeded 
to  work  the  three  practice  items.  They  were  given  an  opportunity  to  ask 
questions  before  continuing  with  the  main  part  of  the  experiment. 

The  pretest  consisted  of  ten  RFB  items,  nine  NRFB  items  and  eight 
almanac  items.  A  questionnaire  comprised  the  last  two  pages  of  the  experi¬ 
mental  materials.  The  first  page  of  questions  asked  the  subjects  to  evaluate 
the  test  items  for  validity  and  realism.  They  were  requested  to  evaluate 
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the  bag  and  chip  problems  as  a  group,  the  NRFB  problems  Individually, 
and  the  almanac  items  as  a  group.  The  responses  received  are  sunsnarixed 
in  Figure  5.  (As  can  be  seen  from  this  figure,  not  all  of  the  participants 
completed  the  questionnaire.)  The  consensus  of  the  respondents  seemed 
to  be  that  the  problems  were  fairly  realistic  and  the  initial  SOX  prob¬ 
ability  assumption  was  not  difficult  to  accept.  However,  there  was 
considerable  uncertainty  expressed  by  the  subjects  concerning  the  accuracy 
of  their  estimates. 

The  second  questionnaire  page  asked  for  background  information  about 
the  subject's  age,  sex,  major,  minor,  grade  point  average,  credits  in 
probability,  statistics  or  decision  theory  and  years  of  education.  Subjects 
were  also  asked  if  they  thought  there  was  a  way  to  calculate  the  bag  and 
chip  problem  probabilities.  A  place  for  conxnents  was  provided.  The 
questionnaire  is  Included  in  Appendix  B  with  the  other  experimental  materials. 

C.  THE  MAIN  EXPERIMENT 

Because  the  responses  of  the  pretest  subjects  indicated  that  the  RFB, 

NRFB  ant*  almanac  problem  situations  were  reasonably  realistic,  these  items 
were  Judged  suitable  for  inclusion  in  the  main  experiment.  Before  subjects 
were  recruited  for  the  experiment,  however,  an  examination  ot  statistical 
power  was  made.  (See  Appendix  A.) 

Although  the  existing  ten  RFB  items  satisfied  the  minimum  power  require¬ 
ments,  four  additional  RFB  problem  situations  were  Included  in  the  main 
experiment.  These  four  items,  which  were  selected  to  have  true  probabilities 
near  the  two  ends  of  the  (0,1)  range,  were  Included  primarily  to  have  a 
moderating  Influence  on  the  number  of  predicted  probability  estimates  that 
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were  outside  this  interval.  (A  number  of  such  out-of-range  predictions 
were  noted  in  analysis  of  the  pretest  data.) 

Students  from  The  Pennsylvania  State  University  were  recruited  as 
subjects  in  the  experiment  by  means  of  an  advertisement  In  the  student 
newspaper.  This  ad  promised  ten  dollars  for  participation  in  a  two-hour 
"...decision-making  study  Involving  a  simple  pencil  and  paper  task.”  To 
Insure  getting  at  least  25  usable  subjects,  allowing  for  various  types 
of  attrition,  a  total  of  51  respondents  were  recruited.  Surprisingly, 
only  four  of  them  failed  to  appear  at  the  session,  so  a  total  of  47 
subjects  took  part. 

The  experiment  was  administered  in  a  single  group  session  with  subjects 
separated  from  each  other  at  large  tables.  The  session  was  continuously 
monitored  by  two  experimenters. 

After  all  subjects  were  seated,  instructions  were  presented  orally. 
Then,  the  printed  Instructions  and  practice  items  (Appendix  B)  were  distri¬ 
buted.  Each  subject  read  the  instructions  and  worked  on  the  practice  items 
independent ly . 

To  insure  understating  of  the  procedure,  the  group  was  asked  for  a 
show  of  hands  for  each  of  the  practice  Items  (e.g.,  "Raise  your  hand  if 
your  estimate  for  Example  A  was  greater  than  50X.")  For  Example  C  (an 
RFB  problem)  no  one  reported  an  estimate  less  than  50X  (l.e.,  a  posterior 
probability  in  the  wrong  direction).  Had  there  been,  further  instructions 
would  have  been  given. 

Subjects  were  Instructed  to  work  the  problems  in  sequence,  to  check 
their  work  when  finished,  to  make  sure  all  items  were  answered,  to  fill 
out  the  quest ionnaire  at  the  end,  and  to  wait  until  everyone  had  finished. 
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The  experimental  materials,  which  were  stapled  in  the  upper  left  hand 
corner,  were  then  distributed  and  the  experiment  began.  The  sets  of 
materials  were  identified  only  by  a  serial  number  on  every  page  to 
insure  that  pages  could  be  collated  should  any  set  come  apart.  Anonimity 
was  thus  preserved,  an  assurance  the  subjects  had  been  given  during  the 
Instruction  period. 

All  subjects  finished  within  the  two  hours  allotted,  with  some 
early  finishers  having  to  wait  fifteen  mmnutes  or  more.  All  experimental 
materials  were  then  collected  and  each  subject  was  paid  for  participating. 
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IV.  DATA  ANALYSIS 


The  fundamental  assumption  on  which  this  study  was  baaed  la  that 
it  should  be  possible  to  construct  (for  each  subject)  a  predictive 
equation,  derived  from  estimates  made  for  RFB  problem  situations,  by 
which  assessments  of  NRFB  probabilities  may  be  calibrated.  The  statis¬ 
tical  procedure  for  testing  this  assumption  is  outlined  in  Appendix  A. 
Although  the  probability  assessments  of  the  subjects  in  both  the  pretest 
and  the  main  experiment  were  subjected  to  statistical  analysis,  only 
the  results  of  the  main  experiment  are  discussed  because  of  the  prelimin¬ 
ary  nature  of  the  pretest. 

As  the  first  step  in  data  analysis,  the  probability  assessments 
were  subjected  to  a  preliminary  examination  aimed  at  uncovering  any 
anomalies.  After  this  examination  was  completed,  regression  techniques 
were  used  to  obtain  calibration  equations  for  predicting  true  probabilities 
from  estimated  probabilities.  These  equations,  In  conjunction  with  the 
experimental  data,  were  then  used  to  test  the  study  assumption  by  means  of 
analysis  of  variance  (ANOVA).  The  following  sections  describe  these  three 
phases  of  data  analysis. 

A.  PRELIMINARY  EXAMINATION  OF  THE  DATA 

Before  any  predictive  (l.e.,  calibration)  equations  were  estimated, 
the  experimental  data  was  subjected  to  a  preliminary  examination.  This 
examination,  which  waa  based  to  a  large  extent  on  various  data  plots, 
was  conducted  primarily  to  check  for  any  anomalies  within  the  probability 
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assessments.  Immediately  obvious  was  one  subject  who  had  estimated  SOX 
ss  the  probability  for  every  one  of  the  problem  situations.  Because  of 
this,  the  subject's  responses  were  discarded,  so  that  subsequent  data 
analysis  was  based  on  a  total  of  46  subjects. 

The  data  examination  also  revealed  that  there  were  extensive  Individual 
differences  in  the  subjects'  assessments.  In  many  instances  probability 
estimates  were  found  to  be  at  opposite  ends  of  the  probability  continuum 
for  the  same  Item.  This  was  not  In  Itself  surprising,  but  while  many  of 
the  discrepant  responses  may  well  have  been  due  to  real  and  consistent 
differences  among  subjects'  probability  perceptions,  they  may  also  have 
been  the  result  of  carelessness  in  responding.  This  possibility  is  dis¬ 
cussed  in  more  detail  in  the  following  sections. 

1.  RFB  Reversals 

In  both  the  pretest  and  the  main  experiment  the  subjects  were  cautioned 
to  pav  particular  attention  to  the  conditions  specified  in  each  RFB  problem 
situation.  This  seemed  advisable  because  although  the  bag  and  chip  prob¬ 
lems  were  all  of  the  same  format,  the  coaq>oaltlon  of  the  hags  varied  from 
Item  to  Item,  as  did  the  Identity  of  the  bag  which  was  to  be  considered  In 
making  the  probability  estimate  response.  If  a  problem  asked  for  the  prob¬ 
ability  of  bag  "B"  but  the  aubject  were  to  make  an  estimate  Instead  with 
respect  to  bag  ”R" ,  a  serious  distortion  would  reault.  Such  responses 
may  be  referred  to  as  reversals.  An  RFB  reversal  response  may  be  defined 
by  comparing  the  subject's  estimate  with  the  true  probability.  Specifically, 
a  reversal  la  either  a  response  which  is  greater  than  SOX  when  the  true 
probability  is  less  than  50X,  or  a  response  which  is  leas  than  50Z  when  the 
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true  probability  is  greater  than  50Z. 

Reversals  were  only  a  minor  factor  in  the  pretest.  Fourteen  of  the 
nineteen  pretest  subjects  were  found  to  have  no  RFB  reversals  whatsoever, 
while  for  three  of  the  five  who  did  have  reversals,  the  effect  was  insignifi¬ 
cant.  At  the  time  it  was  thought  that  the  reversals  might  have  resulted 
from  time  pressure,  and  that  with  ample  time  the  number  of  reversals  would 
be  minimized.  To  guard  against  reversals  the  procedure  in  the  main  experi¬ 
ment  was  revised  to  emphasize  in  the  instructions  that  the  conditions  for 
each  RFB  item  should  be  considered  carefully.  A  poll  of  the  subjects  after 
completing  the  practice  items  did  not  indicate  any  apparent  need  to  further 
reinforce  the  instructions. 

However,  after  the  data  from  the  main  experiment  was  examined,  it  was 
apparent  that  reversals  could  cause  a  serious  analysis  problem.  Although 
only  6t  of  the  644  total  RFB  estimates  were  reversals,  almost  half  of  the 
participants  made  at  least  one  reversal  response.  Because  reversals 
could  potentially  have  a  significant  effect  on  the  outcome  of  the  experi¬ 
ment,  the  resulting  data  was  subsequently  analyzed  two  ways.  In  one  analysis 
the  responses  were  taken  as  given,  while  in  the  other  analysis  each  RFB 
reversal  was  "revised"  by  subtracting  the  estimated  probability  from  unity. 

The  effects  of  a  reversal  may  he  seen  by  examining  Figure  6,  which 
presents  a  plot  of  true  probability  against  estimated  probability  for 
one  of  the  subjects  in  the  main  experiment.  As  can  be  seen,  the  point 
1.88,  .015)  seriously  distorts  what  la  otherwise  a  reasonable  linear  relation¬ 
ship.  Revision  of  the  reversal  corrects  this  distortion. 

As  an  explanation  of  the  reversals,  s  fatigue  effect  might  be  postu¬ 
lated.  However,  although  there  were  slightly  more  reversals  among  the 
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FIGURE  6:  An  Example  of  an  RFB  Reversal 


last  seven  RFB  Items  than  among  the  first  seven,  there  were  more  among 
the  first  four  items  than  among  the  last  four.  It  is  interesting  to  note 
that  there  were  23  reversals  of  items  which  had  true  probability  less 
than  50X  and  only  13  reversals  of  items  in  which  the  true  probability  was 
greater  than  50X,  perhaps  indicating  a  tendency  to  estimate  the  probability 
of  the  most  likely  hypothesis.  Nonetheless,  these  results  are  not  statis¬ 
tically  significant.  Since  the  reversal  patterns  cannot  be  satisfactorily 
explained  by  other  assumptions,  it  may  be  that  lack  of  attention  and  care¬ 
lessness  were  the  causes. 

2.  NRFB  Reversals 

Just  as  it  is  possible  for  a  subject  to  give  a  response  to  an  RFB 
item  which  is  in  a  sense  the  reverse  of  that  subject's  underlying  esti¬ 
mate,  it  is  conceivable  that  a  response  to  an  NRFB  item  might  also  be  a 
"reversal."  In  other  words,  the  subject  might  be  estimating  the  probability 
of  the  complementary  hypothesis.  However,  there  is  no  "true"  probaollity 
against  which  to  compare  NRFB  estimates  as  there  ia  for  the  RFB  book-bag 
and  poker  chip  problems  used  in  this  study.  Although  it  is  possible  to 
use  the  arithmetic  mean  over  all  subjects  as  the  criterion  for  defining 
an  NRFB  reversal,  the  justification  is  much  more  tenuous.  Nevertheless, 
if  NRFB  reversals  were  defined  in  this  way,  a  conaiderable  number  of  NRFB 
estimates  could  be  classified  as  reversals. 

B.  ESTIMATION  OF  PREDICTIVE  EQUATIONS  f  ,, 

I  4 

In  the  investigation  of  the  functional  (i.e.,  predictive)  relationship 
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between  true  and  estimated  probability,  attention  was  restricted  to 
relationships  which  are  intrinsically  linear  [Draper  and  Smith  (1966)] 
and,  more  specifically,  which  are  or  may  be  transformed  to  a  low-order 
polynomial.  Thus,  the  added  complexities  of  nonlinear  estimation  were 
avoided.  This  restriction  did  not  impose  extreme  limitations;  it  per¬ 
mitted  consideration  of  the  dependent  variable  as  any  polynomial  function 
of  the  independent  variable  and  allowed  the  use  of  any  chosen  transfor¬ 
mations  of  the  true  and  estimated  probabilities. 

In  the  analysis  of  the  data,  attention  was  concentrated  on  prob¬ 
ability  and  log  odds  variables.  Both  p  (true  probability)  and  ln[p/(l  -  p) ] 
were  plotted,  as  dependent  variables,  against  r  (estimated  probability)  and 
ln(r/(l  -  r))  as  independent  variables.  Linear,  quadratic  and  cubic 
regression  equations  were  fitted  for  all  four  combinations  of  these 
dependent  and  independent  variables  for  each  subject. 

The  best  functional  realtionships  were  obtained  using  true  probability 
versus  estimated  probability  or  true  log  odds  versus  estimated  log  odds. 
Mixing  probability  with  log  odds  provided  less  useful  calibration  equations, 
as  Judged  by  observed  F  and  R*"  values.  In  general,  linear  equations  were 
found  to  provide  a  reasonable  fit,  since  linear  terms  were  usually  highly 
significant  while  quadratic  and  cubic  terms  were  nonsignificant.  However, 
the  predictive  ability  (measured  by  R^  and  residual  mean  square)  of  the 
regression  equation  was  disappointing  for  those  subjects  who  had  RFB 
reversals.  When  these  reversals  were  revised,  major  improvement  was  noted. 
For  example,  for  probabi 1 i t les  as  variables,  the  minimum  R‘  value  was  .755  . 
Figure  7  lists  the  observed  F  and  R  values  for  the  linear  regression 
equations  cor respond ing  to  each  of  the  46  subjects  based  on  the  revised 
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FIGURE  7  :  Observed  F  and  R  Values  for  Linear  Regression  Equation  (Revised  RFB  Data) 


As  expected,  there  could  be  instances  when  a  predicted  probability 
computed  from  a  regression  equation  would  fall  outside  the  (0,1)  range 
over  which  probabilities  are  defined.  To  lessen  the  effect  of  out-of¬ 
range  predicted  probabilities,  these  probabilities  could,  of  course,  be 
constrained  to  fall  in  the  (0,1)  range  by  setting  values  larger  than 
one  equal  to  one  and  values  less  than  zero  equal  to  zero. 

In  view  of  the  previous  discussion,  three  calibration  equations 
for  probability  prediction  were  considered  for  each  subject: 

(1)  an  equation  based  on  the  original  RFB  estimates, 

(2)  an  equation  based  on  the  revised  (reversal-corrected)  RFB 
est imates , 

and  (3)  an  equation  based  on  revised  RFB  estimates  with  predicted 
values  subject  to  range  constraints. 

In  addition,  two  equations  for  log  odds  prediction  based  on  a  conversion 
ot  the  probability  estimates  to  log  odds  were  considered:  one  derived 
from  the  original  unrevised  RFB  estimates,  and  the  other  based  on  revision 
of  RFB  reversals. 

By  using  R‘  as  the  criterion,  it  was  determined  that  some  subjects' 

RFB  estimates  were  consistently  much  more  highly  correlated  with  the  true 

2 

probability  than  others.  There  were  fourteen  subjects  for  whom  the  R 
value  based  on  revised  RFB  estimates  was  .90  or  larger.  Calibration 
equations  were  therefore  obtained  for  this  group  of  "most  consistent 
estimators"  as  well  as  for  the  full  complement  of  46  main  experiment  subjects 
and  for  the  subset  of  24  subjects  having  no  RFB  reversals. 


C.  EXPERIMENTAL  RESULTS 

Probability  land  log  odds)  predictions  were  based,  of  course,  on 
the  linear  regression  equations  derived  from  estimated  probabilities  in 
RFB  problem  situations.  Predictions  were  made  for  the  primary  set  of 
Interest  (the  nine  NRFB  items)  and  for  the  eight  almanac  problems.  As 
a  means  of  validating  the  fundamental  concept  of  this  study,  predictions 
were  also  made  using  linear  regression  equations  derived  from  the  first 
ten  of  the  RFB  items  (an  arbitrary  selection)  to  predict  values  for  the 
remaining  four  RFB  items.  The  resulting  predictions  in  each  of  these  three 
cases  were  subjected  to  the  hypothesis  tests  outlined  in  Appendix  A. 


1,  Predictions  of  NRFB  Probabilities 

To  examine  the  assumption  that  assessments  of  NRFB  probabilities  may 
be  calibrated  by  means  of  an  RFB-based  predictive  equation,  the  experi¬ 
mental  results  were  statistically  evaluated  by  ANOVA  for  each  Individual 
N'RFB  problem  across  all  subjects  and  in  an  overall  evaluation  across  all 
NRFB  problems  and  all  subjects.  These  Individual  and  overall  analyses 
were  also  performed  for  a  subset  of  the  24  subjects  without  RFB  reversals 
and  for  the  14  most  consistent  estimators.  In  addition,  three  different 
data  sets  were  considered.  The  first  was  based  on  the  subjects'  estimates 
as  given,  while  in  the  second  the  RFB  reversals  were  revised.  The  third 
was  based  on  revision  of  both  RFB  reversals  and  NRFB  reversals. 

Figure  8  shows  the  results  of  the  overall  statistical  tests.  Large 


F-ratios  and  low  probabilities,  normally  considered  desirable  as  a  research 
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result,  here  Imply  significantly  large  differences  between  predicted 
probabilities  (or  log  odds)  for  different  subjects,  which  tends  to 
reject  the  assumption  that  the  RFB-based  models  would  result  in  pre¬ 
dictions  which  were  all  unbiased  estimates  of  the  same,  unknown, 
probability  for  any  given  problem.  Thus,  in  the  current  situation, 
small  F  values  and  high  probabilities  support  the  underlying  assumption. 

From  an  examination  of  the  figure,  it  can  be  seen  that  overall 
agreement  between  predictions  of  the  NRFB  probabilities  was,  in  general, 
unsatisfactory.  With  all  reversals  revised  and  range-constrained  prob¬ 
abilities  used,  the  underlying  assumption  would  not  have  been  rejected 
for  the  set  of  all  46  subjects  and  for  the  subset  of  24  subjects  without 
RFB  reversals.  However,  these  results  were  based  on  the  revision  of 
S'RFB  reversals,  which  seems  to  be  an  extremely  tenuous  procedure. 

2.  Prediction  of  Almanac  Probabilities 

Almanac  items  of  the  type  employed  in  this  experiment  were  of  some 
Interest  because,  although  they  did  not  have  a  general  information  format 
similar  to  the  RFB  and  NRFB  problems,  they  did  have  true  probabilities 
associated  with  them.  Thus,  the  items  were  Included  in  this  experiment 
to  provide  supplementary  information. 

Subjects’  estimates  correlated  poorly  with  the  true  probabilities 
for  most  of  the  almanac  items.  More  importantly,  as  Figure  9  indicates, 
RFB-derived  predictive  models  provided  an  unstable  basis  for  calibrating 
probability  assessments  in  the  almanac  situations.  This  result  was 
expected,  however,  because  of  the  dissimilarity  between  the  almanac  and 
RFB  format.  While  the  RFB  situations  were  designed  as  two-hypothes is 
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problems  where  a  prior  probability  of  501  was  to  be  revised  in  the  light 
of  new  data,  the  almanac  problems  asked  for  a  direct  probability  assess¬ 
ment  . 

3.  Prediction  of  RFB  Probabilities 

Although  the  failure  of  the  RFB-based  model  when  applied  to  the 
almanac  items  was  not  surprising,  their  poor  performance  on  the  NRFB 
problems  cast  considerable  doubt  on  the  overall  validity  of  the  suggested 
calibration  approach.  Of  course,  success  or  failure  in  predicting 
NRFB  probabilities  using  models  derived  from  RFB  probability  estimates 
may  depend  as  much  on  the  similarity  of  the  two  estimation  tasks  as  it 
does  on  the  development  of  a  satisfactory  calibration  equation. 

In  order  to  gain  insight  into  this  matter,  additional  analyses  were 
performed  on  the  reversal-corrected  RFB  estimates  obtained  in  the  experi¬ 
ment.  The  first  ten  RFB  items  were  used  to  derive  equations  of  the 
subjects'  estimation  behavior,  and  these  were  then  used  in  adjusting 
estimates  made  on  the  last  four  RFB  items  to  arrive  at  predicted  values. 
Therefore,  the  data  used  for  constructing  the  equations  was  Independent 
of  that  used  for  testing  them.  Since  the  task  was  identical  for  each 
group  of  Items,  the  only  variable  left  to  explain  failure  would  be  an 
inadequate  calibration  equation. 

As  Figure  10  shows,  good  results  were  obtained  for  probability  models 
and  for  log  odds  models.  The  best  results  were  obtained  when  predicted 
probabilities  were  constrained  to  the  (0,1)  range.  In  view  of  the  almost 
complete  lack  of  success  with  the  NRFB  predictions,  these  results  were 
unexpected. 
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FIGURE  10:  Observed  F-ratios  for  Overall  Teat  on  Four  RFB  Probleas 

(Corresponding  p-values  in  Parentheses) 


It  should  be  noted  that  the  tests  Indicate  relatively  small  vari¬ 
ability  between  the  set  of  predictions  obtained  for  a  given  problem  by 
means  of  the  RFB-derived  models.  Thus,  they  strongly  support  the 
proposition  that  for  any  given  RFB  problem  the  models  result  in  unbiased 
estimates  of  the  same  population  parameter. 

However,  the  tests  provide  no  comparison  of  predictions  with  true 
values.  Figure  11  graphically  Indicates  (for  the  probability  models) 
the  average  estimates,  the  average  predictions,  and  the  true  probability 
values  for  each  of  the  four  RFB  problems.  As  can  be  seen,  in  all  except 
the  last  problem  the  predictions  obtained  from  application  of  the  RFB- 
based  regression  equations  were  ouch  closer  to  the  true  values  than  were 
the  estimates  obtained  directly  from  the  subjects. 
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V.  SUMMARY  AND  DISCUSSION 

The  experiment  conducted  in  this  study  had  as  its  main  objective 
the  investigation  of  the  hypothesis  that  probability  estimates  made 
for  a  set  of  "book-bag  and  poker  chip"  relative  frequency  base  (RFB) 
tasks  could  be  used  to  derive  satisfactory  models  for  providing  accurate 
probability  assessments  for  a  set  of  general  information  non-relative 
frequency  base  (NRFB)  problem  situations.  A  fairly  comprehens ive  set  of 
predictive  models  was  employed,  some  of  which  proved  to  he  unsatisfactory 
and  were  eliminated  from  further  consideration.  In  fact,  it  was  never 
possible  to  make  satisfactory  predictions  of  NRFB  probabilities  when  the 
subjects'  raw  RFB  probability  estimates  were  used  in  constructing  a  model. 

Some  improvement  was  obtained  by  revising  all  RFB  estimation  reversals 
and  constraining  the  range  of  all  predicted  probabilities.  However,  It 
was  not  until  NRFB  reversals  were  also  revised  that  any  satisfactory 
results  were  obtained.  Unfortunately,  any  proposed  Justification  for  this 
last  step  is  questionable. 

Despite  this  lack  of  success,  there  is  strong  evidence  that  the  under¬ 
lying  approach  may  be  valid  If  the  probability  estimation  tasks  to  which 
a  predictive  model  is  applied  are  identical  to  those  used  in  model  construc¬ 
tion.  This  evidence  Is  provided  by  the  results  of  using  estimates  made 
on  the  first  ten  of  the  RFB  problems  to  construct  models  which  were  then 
used  to  make  predictions  for  the  remaining  four  RFB  items.  The  predictions 
thus  made  were  found  to  support  the  suggested  approach. 

It  should  he  recalled  that  the  RFB  and  NRFB  problem  situations  were 
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specifically  designed  to  be  of  parallel  construction.  Of  necessity, 
however,  the  data  and  hypotheses  in  the  RFB  problems  differed  from  those 
in  the  NRFB  problems.  If  the  subjects  perceived  the  NRFB  problems  as 
inherently  distinct  from  the  RFB  problems,  this  might  account  for  the 
failure  of  RFB-based  models  to  provide  adequate  predictions  of  NRFB 
probabilities. 

Another  explanation  for  the  experimental  results  might  be  that  in 
real-world  situations  (such  as  the  NRFB  problems  purport  to  be)  probabil¬ 
ities  are  assessed  on  the  basis  of  "common  sense"  and  experience  in 
dealing  with  past  problems.  Each  individual  brings  to  each  problem  a 
different  experiential  framework  (the  result  of  many  events  and  their 
interactions)  which  causes  him  to  approach  the  problem  from  his  own 
perspective.  Each  individual's  experiential  framework  will  affect  the 
cognitive  processes  as  he  refers  to  the  framework  in  estimating  a  probabil¬ 
ity.  The  situation  is  further  complicated  if  the  subject  perceived  each 
NRFB  problem  as  being  so  different  from  every  other  NRFB  problem  that  it 
was  necessary  for  him  to  approach  each  task  from  a  different  viewpoint 
within  his  experiential  framework. 

On  the  other  hand,  in  the  case  of  the  "book-bag  and  poker  chip"  RFB 
problems,  more  similarity  of  experience  would  be  expected  on  a  problem-to- 
problem  basis.  Furthermore,  a  uniformity  of  approach  to  these  problems  was 
somewhat  guaranteed  by  the  fact  that  they  were  of  identical  structure. 
Therefore,  even  without  a  well-defined  mathematical  procedure  to  follow, 
it  would  be  expected  that  an  individual,  once  having  formed  an  empirical 
procedure  of  his  own  geared  to  the  particular  problem-solving  task,  would 
solve  all  the  identically-structured  RFB  problems  according  to  that  procedure. 
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In  summary,  it  is  reasonable  to  conclude  from  this  experiment  that 
models  derived  from  RFB  estimates  can  provide  satisfactory  adjustments  of 
probability  estimates  made  for  another  set  of  tasks,  if  these  tasks  have 
the  identical  structure  of  the  RFB  problem.  It  should  be  noted  that 
although  identical  structure  has  been  shown  to  be  a  sufficient  condition 
for  successful  performance  of  the  suggested  approach,  this  does  not  mean 
that  it  is  a  necessary  condition.  If  it  were,  the  approach  would,  of 
course,  be  of  severely  limited  applicability  in  real-world  situations, 
since  few  decision-making  tasks  have  a  "book-bag  and  poker  chip"  format. 
However,  it  would  be  unwise  to  discount  the  feasibility  of  developing  a 
practical  RFB-based  model  for  adjusting  real-world  probability  estimates 
until  the  approach  were  tested  with  participants  and  problems  in  a  field 
of  common  expertise  (such  as  intelligence  analysis  or  weather  forecasting, 
for  example)  where  more  uniformity  of  background  and  experience  exists. 
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APPENDIX  A:  STATISTICAL  CONSIDERATIONS 


As  outlined  in  section  II,  it  was  assumed  that  a  functional 
relationship  between  true  probability  and  estimated  probability  can 
be  identified  for  any  given  individual.  It  was  also  assumed  that  this 
relationship  could  be  estimated  by  means  of  regression  techniques. 

Because  the  dependent  variable  would  be  true  probability  (or,  perhaps, 
a  transformation),  some  questions  are  raised  if  the  overall  regression 
situation  is  viewed  from  a  classical  statistical  standpoint. 

In  a  classical  statistical  framework,  the  true  probability  p  for 
any  problem  is  a  constant  value  and  therefore  does  not  have  a  probability 
distribution  (except,  of  course,  a  degenerate  one).  Because  the  dependent 
variable  under  consideration  is  not,  in  fact,  a  random  variable,  the 
regression  of  true  probability  on  estimated  probability  has  no  meaning 
from  a  classical  point  of  view.  [See  Keeping  (1962),  for  example.)  If, 
however,  the  problem  is  viewed  in  a  Bayesian  framework,  it  can  be  verified 
that  the  true  probability  p  does  have  a  probability  distribution. 

Consider,  for  example,  the  class  of  problems  which,  for  person  i. 


give  rise  to  the  same  estimated  probability  r^  .  In  general,  problems 
within  this  class  have  different  true  probabilities.  Thus,  in  a  Bayesian 


sense,  the  true  probability  of  any  problem  within  this  class  is  a  random 


variable.  Furthermore,  as  this  argument  is  repeated  for  all  values  of  r^ 
in  the  interval  (0,1),  it  is  reasonable  to  assume  that  the  expected  value 


of  p  is  conditional  on  r^  •  Therefore,  in  this  framework,  the  underlying 
assumptions  of  regression  are  satisfied.  Attention  may  now  be  turned  to 
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determination  of  the  functional  relationship  between  true  probability 
and  estimated  probability. 

A.  FUNCTIONAL  RELATIONSHIPS 

Given  the  regression  situation  with  dependent  variable  p  (true 
probability)  and  independent  variable  r  (estimated  probability),  there 
are  two  major  approaches  to  postulating  the  mathematical  form  of  the 
relationship  between  the  two  variables.  The  first  is  based  on  apriori 
assumptions;  the  second  is  based  on  examination  of  the  scatter  plot 
between  the  two  variables  or  their  transforms.  Both  approaches  have 
been  adopted  in  this  study. 

The  simplest  functional  relationship  between  two  variables  y  and  x 
is,  of  course,  a  straight  line.  The  regression  model  for  this  relation¬ 
ship  would  be 


E(y)  -  a  +  6x  . 

This  is,  in  essence,  the  relationship  assumed  by  Sibley  (1974)  in  con¬ 
junction  with  the  postulated  realism  line.  However,  because  a  linear 
relationship  does  not  place  restrictions  on  the  predicted  values  of  y  , 
the  logistic  function  has  been  suggested  [Warner  (1963),  Theil  (1967)] 
as  a  more  appropriate  model  when  probabilities  are  involved. 

The  logistic  functlorf,  which  restricts  predicted  y  values  to  (0,1), 
takes  the  form 

E(y)  •  (1  +  exp [ -(a  +  8x)])  *. 

If,  for  example,  the  observed  data  in  Figure  1  of  Beach  and  Wise  (1969) 


-47- 


is  examined,  it  can  be  seen  that  the  relationship  between  estimated 
probability  r  and  true  probability  p  appears  to  be  reasonably  well- 
represented  by  a  fitted  logistic  function  of  the  for® 

r  -  {1  +  exp [- (a  +  bp)]}  l.  (1) 

An  "eyeball"  examination  suggests  that  values  of  a  -  -1.0  and  b  -  2.0 
would  provide  a  reasonably  good  fit  to  this  data.  (Of  course,  actually 
fitting  a  regression  equation  to  the  data  would  provide  more  accurate 
values  of  a  and  b  .) 

The  relationship  between  r  and  p  given  in  (1)  may  be  expressed 
equivalently  (after  a  few  algebraic  steps)  in  the  form 

P  -  c  +  d  •  In [r/  (1  -  r )  ]  (2) 

Thus,  it  is  reasonable  to  investigate  the  use  of  p  as  dependent  variable 
and  ln[r/(l  -  r)]  as  independent  variable,  since  it  may  be  that  the  result¬ 
ing  regression  equation  in  this  case  is  linear. 

Other  research  [Peterson,  Schneider  and  Miller  (1969),  for  example] 
suggests  the  relationship 

BLLR  -  k  •  SLLR  (3) 


where : 


BLLR  denotes  Bayesian  log  likelihood  ratio 
SLLR  denotes  subjective  log  likelihood  ratio 
and  k  is  a  constant. 

When  the  prior  probability  of  each  of  the  two  hypotheses  under  considera¬ 
tion  is  50%,  BLLR  is  equivalent  to  ln[p/(l  -  p)J  and  SLLR  is  equivalent 
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to  ln[r/(l  -  r)l,  so  that  (3)  may  be  written  as 

ln[p/(l  -  p)]  -  k  •  ln(r/(l  -  r)J  .  (4) 

Thus,  this  transformat  ion  of  p  and  of  r  should  be  considered. 

Of  course,  in  actually  fitting  a  relationship  as  indicated  in 
(4),  a  constant  term  would  be  used  so  that  the  resulting  equation 
would  be  of  the  form 

ln[p/(l  -  p)]  -  h  +  k  •  ln[r/(l  -  r)J  .  (5) 

It  should  be  noted  that  this  equation  is  algebralcly  equivalent  to 

p  *  {l  ♦  expl-(a  +  b  •  ln[r/(l  -  r )  ] )  ] } 

which  is  the  form  of  the  suggested  logistic  function,  but  in  terms  of  a 
transformed  independent  variable. 

In  view  of  the  foregoing  discussion,  It  appears  that  attention  should 
be  focused  on  the  untranaforsu  variables  p  and  r  and  on  the  transformed 
variables  ln(p/(l  -  p)]  and  ln(r/(l  -  r ) ]  .  Specifically,  the  following 
three  combinations  of  dependent  and  Independent  variables  are  of  primary 
interest  in  the  examination  of  the  functional  relationship  between  esti¬ 
mated  and  true  probability: 


P  ln[r/(l  -  r ) ) 

In (p/(l  -  p) )  ln|r/(l  -  r) J 

The  untransformed  variables  are,  of  course,  measured  on  the  original 
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scale  over  the  interval  (0,1).  The  transformed  variables  are,  because 
of  the  uniform  prior,  equivalent  to  log  likelihood  ratio  and  to  log  odds, 
which  have  been  used  extensively  in  probability  studies  [Peterson  and 
Swenson  (1968),  Peterson,  DuCharme,  and  Edwards  (1968),  Johnson  (1974), 
for  example).  As  Edwards  (1966a)  points  out,  log  likelihood  ratio  and 
log  odds  often  tend  to  be  the  moat  useful  variables  for  analysis  of  prob¬ 
ability  estimation  experiments  which  Involve  discrete  hypotheses. 

B.  HYPOTHESIS  TESTING 

Assume  that  a  total  of  I  subjects  participate  in  the  experiment, 
with  each  subject  providing  an  estloMted  probability  for  each  of  M 
relative  frequency  base  (RFB)  problems  and  K  nonrelative  frequency  base 
(NRFB)  problems.  Assuaw  that,  based  on  the  H  RFB  problems,  a  fitted 
equation  p  -  f((r)  Is  found  to  provide  a  good  approximation  to  the 
relationship  between  true  probability  p  and  estimated  probability  r  for 
subject  1  . 

As  discussed  previously,  either  the  true  probability  p  or  some 
transformat  ion  of  it  oav  be  used  as  the  dependent  variable.  For  the 
remainder  of  this  section,  however,  the  discussion  assumes  the  untrans- 
foraed  probability  p  to  be  the  dependent  variable.  This  assumption  Is 
made  only  to  simplify  the  discussion  and  results  in  no  lack  of  generality, 
since  a  similar  analysis  holds  for  any  transformation  of  p  . 

For  the  kt h  NRFB  problem  (where  the  true  probability  is,  of  course, 
unknown),  consider  the  probabilities  ,...,r  estimated  by  the  I 

A 

subjects.  The  functions  f  (r  )  for  1  “  1,...,I  may  be  used  to  provide 
"corrected"  estimates  of  the  true  (unknown)  probability  p^  .  If  the 


use  of  these  functions  to  obtain  corrected  estimates  Is  valid,  then  the 

A  A 

estimates  Pi^»-,,,Pj^  ahould  be  unbiased  estimates  of  p^  .  In  that  event, 

A 

the  variability  among  the  p^'s  would  be  totally  accounted  for  by  the 
random  error  inherent  in  any  atatlatlcal  quantity. 

A 

In  general,  however,  each  p  ,  is  an  unbiased  estimate  of  some 

lk 

(unknown)  probability  p^  .  If  all  the  p^'e  are  not  equal  (l.e.,  the 

functions  do  not  result  in  unbiased  estimates  of  the  same  probability  p  ), 

k 

A 

the  the  variability  among  the  p^'s  will,  in  addition  to  random  error, 
be  attributed  to  the  true  differences  in  the  p^'s  •  Thus,  th-  "corrected" 
estimates  may  be  examined  in  a  hypothesis  testing  framework  with  the  follow¬ 
ing  hypotheses: 


"o'  "lk*  •  Plk 

H  :  All  the  p  ’a  are  not  equal. 

1  ik 

It  was  noted  previously  that  the  fitted  function  f^fr)  provided  the 

estimate  p^  .  In  addition,  the  actual  fitting  of  the  function  provides 

-2 

an  estimate  0.^  of  the  underlying  variance.  This  estimate  is,  of  course, 

given  by  the  residual  mean  square.  If  f^(r)  is  assumed  to  be  a  two- 

'2 

parameter  function,  0  ^  will  have  M  -  2  degrees  of  freedom. 

A  test  of  hypothesis  against  hypothesis  may  be  based  on  the 
F-stat 1st ic 


F  -  MSB/MSE 


where  MSB  Is  toe  between-subj ects  mean  square 


MSB  -  1  Up  -  <Ep  /I)  I 

I  -  1  i  lh  i  lk 


51 


and  MSE  la  the  error  mean  square  formed  from  pooling  the  estimates  0 


HSE 


2  2 

Assuming  that  each  0^  -  0  (a  reasonable  assumption  In  view  of  the 
robustness  of  the  F-teat  to  inequality  of  variance),  the  F-statistic  has 
an  F-distributlon  with  1-1  and  I(M  -  2)  degrees  of  freedom  if  the  null 
hypothesis  Is  true.  Thus,  the  test  would  be 


( 


Reject  H  if  F  >  F 

0  a, I  -  1 


I(M 


2) 


I  Do  not  reject  H  if  F  <  F 
f  0  ~  a,  I  -  1,  I(M  -  2) 

where  F  denotes  the  upper  o  point  of  the  F-dlstribution 

a. I  -  l,  KM  -  2) 

with  I  -  1  and  I(M  -  2)  degrees  of  freedom. 

Although  It  Is  Important  to  examine  each  particular  NRFB  problem 
Individually,  It  would  be  desirable  to  conduct,  for  the  K  NRFB  problems, 
one  overall  test  of  the  hypotheses: 


H  : 
0 


IK 


IK 


Hq  Is  not  true 


The  use  of  external  estimates  0  precludes  a  multivariate  test  because 
not  enough  Information  is  available  to  permit  calculation  of  the  cross- 


product  ten 


required  In  the  matrix  due  to  experimental  error.  (See 


Winer  (1971)).  A  reasonable  alternative  Is  to  analyze  the  data  as  arising 
from  a  repeated  measures  design.  A  d lagramnat ic  view  of  the  situation  and 
the  accompanying  analysis  of  variance  are  given  In  Figure  Al. 

As  can  be  seen  from  the  figure,  the  usual  repeated  measures  design 
does  not  permit  a  teat  of  between-subject  variation,  which  Is  usually  not 
of  Interest.  In  the  current  situation,  however,  this  variation  is  of  Interest 
and  may  be  tested  against  the  pooled  error  mean  square 


KSE  •  1  '2 

«  S"u 


2 

It  should  be  noted  that  for  a  given  subject,  each  0  (k  ■  1 . K)  is  based 

lk 

on  the  same  M  data  points.  Thus,  MSE  real  1st ically  has  I (M  -  2)  rather  than 
IK(M  -  2)  degrees  of  freedom. 

From  Figure  Al  It  can  be  seen  that  hypothesis  HQ  (that  the  "correction" 

2  2 

procedure  Is  valid)  is  equivalent  to  <1.  •  0  and  a„  •  0  .  Thus,  to  test  H  , 

SR  0 

MSB  and  MSR  must  be  treated  against  MSE.  However,  since  It  Is  desired  to 
place  In  Jeopardy,  It  Is  not  enough  Just  to  set  up  a  test  with  a  given 
Type  I  error.  A  relatively  small  Type  II  error  or  large  power  (i.e.,  a 
good  chance  of  detecting  real  differences)  within  reasonable  limitations 
on  number  of  subjects  and  problems  Is  also  required. 


C.  STATISTICAL  POWER 

If  the  experimental  situation  outlined  in  the  previous  sections  is 

reviewed,  it  will  be  seen  that  departures  from  the  null  hypothesis  should 

2 

be  reflected  primarily  In  0^  (which  measures  the  differential  performance 

2 

of  subjects  on  problems),  rather  than  In  (which  measures  between-subject 
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variability  averaged  over  all  problems).  Thua,  If  the  null  hypothesis  Is 

2 

false.  It  Is  reasonable  to  assume  that  0R  will  be  relatively  large  compared 

2 

to  0S  .  In  view  of  this  situation,  the  exakiaation  of  power  was  focused 

2 

on  the  test  of  0  •  0  . 

It  can  be  shown  (Scheffe  (1959),  Winer  (1971))  that  the  statistic 
F  -  HSR/MSE  is  distributed  as 


[I  ♦  (o^/c  ))F(i  _  1)(K  _  l(H  _  2) 


which  reduces  to  F  If  H-  la  true.  Thua,  the  power 

(I  “  1)(K  •  1)*  I (M  •  2)  y 

of  this  test  depends  on  the  size  of  relative  to  02,  on  the  values  of  I 

(the  number  of  subjects),  K  (the  number  of  N&7B  problems),  and  M  (  the 

mrniber  of  RFB  problems),  and,  of  course,  on  the  significance  level  a 

selected  for  the  test.  From  a  practical  standpoint.  It  is  desirable  that 

2  2 

the  test  be  able  to  detect,  with  reasonable  certainty,  values  of  0  >  o 

Figures  A2  and  A3  exhibit,  for  a  -  .05  and  o  -  .10,  power  curves  for 

2  2 

different  values  of  0  •  0R /o  ,  1,  K,  and  H.  Based  on  these  and  similar 
curves.  It.  was  decided  that  values  of  1  *  25,  K  ■  8,  and  M  •  10  provided 
minimum  requirements. 
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APPENDIX  B:  materials  used  in  the  experiment 


The  following  pages  comprise  the  set  of  materials  used  in  the 
main  experiment.  These  materisls  consist  of  an  Introductory  section, 
examples,  thirty-one  (fourteen  RFB ,  nine  NRFB,  eight  almanac)  problem 
situations,  and  a  questionnaire. 
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DESMATICS,  INC. 


P  O  Box  6)8 

Stott  College,  Po  16801 

Phone  (814)  238-9621 


DECISION-MAKING  STUDY:  INTRODUCTION 


Purpose  of  Study 

The  purpose  of  this  study  is  to  provide  lnforastlon  which  can  be  used 
in  development  of  Improved  decision-making  methods.  The  study  is  being 
conducted  by  Desmatlcs,  Inc.  for  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  under  contract  DABC19-76-C-0045.  The  results 
will  be  published  in  a  report  at  the  end  of  the  contract  later  this  year. 

An  abstract  of  the  report  will  be  published  in  a  future  issue  of  "Government 
Report  Announcements  and  Index,"  a  copy  of  which  may  be  found  in  most  large 
libraries . 

Summary  of  Study 

This  study  is  concerned  with  probability,  which  is  nothing  more  than 
the  chance  that  something  will  or  will  not  happen.  There  are  many  situations 
in  which  the  term  "probability"  is  used.  For  example,  weather  reports  regu¬ 
larly  contain  statements  such  as  "...the  probability  of  rain  is  40 X  tonight 
and  70Z  tomorrow..." 

Suppose  that  a  forecast  of  the  probability  of  rain  in  the  local  vicinity 
is  to  be  made  for  Nay  29,  1977.  Assume  that  historical  weather  data  indicates 


that  for  all  the  May  29  dates  in  previous  years,  there  was  rain  on  half  of 
them  and  no  rain  on  the  other  half.  Based  solely  on  this  historical  data, 
it  would  be  reasonable  to  make  an  initial  forecast  for  Hay  29,  1977,  which 


stated  "....the  probability  of  rain  is  50Z...." 

However,  aa  the  May  29  date  approaches,  new  Information  may  change  this 
forecast.  For  example,  suppose  that  on  May  28  there  were  rain  clouds  in  the 
west,  a  falling  barometer,  and  a  change  of  wind  direction  so  that  prevailing 
winds  were  from  the  west.  Moat  weather  forecasters  (and  most  other  people) 
would  feel  that  the  initial  forecast  of  50Z  probability  should  be  Increased. 

In  most  situations  a  probability  cannot  be  precisely  calculated — only 
estimated.  Of  course,  probability  estimates  made  by  different  people  often 
vary.  This  study  is  aimed  at  determining  the  nature  and  extent  of  this 
variability  in  selected  situations. 

Today  you  will  be  presented  with  a  number  of  problem  situations.  In 
each  one  you  will  be  asked  to  estimate  the  probability  (that  is,  the  chance) 
of  some  specific  thing  happening.  In  making  your  estimates,  try  to  have 
them  reflect  your  judgment  as  accurately  as  possible,  even  if  you  find  it 
difficult  to  decide  on  an  estimate. 

Some  Guidelines  for  Making  Estimates 

When  you  are  estimating  probabilities,  bear  in  mind  that  probability  may 
range  from  OZ  (for  something  which  will  never  occur)  to  100Z  (for  something 
which  Is  certain  to  occur).  Also,  it  may  be  helpful  to  begin  by  mentally 
placing  your  estimate  "in  the  right  ball  park."  For  example,  in  a  certain 
situation  you  might  feel  that  your  estimate  should  bs  somewhere  between  6SZ 
and  80Z.  Then  "sero  in"  on  your  estimate  by  selecting  a  single  value  within 
this  range. 

Please  try  to  make  your  flaal  estimate  so  that  you  feel  it  Is  an  accurate 
reflection  of  your  Judgment.  For  example,  if  you  feel  the  probability  is 
somewhere  between  70Z  and  75X,  do  not  estimate  70X  or  7SZ  solely  because 
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they  are  nice  round  mabera.  Instead  ask  yourself  If  perhaps  s  better 
est taste  might  be  71Z  or  72Z.  When  you  have  aade  your  decision,  write  down 
your  estimate  aa  a  percentage  between  OZ  a:id  100Z. 

Overview  of  Today's  Session 

Airing  today's  session  you  will  be  asked  to  estimate  probabilities  in  a 
few  staple  situations  which  will  be  exp La in ad  in  a  moment.  At  the  end  of 
the  session,  we  would  like  you  to  answer  a  short  questionnaire. 

You  will  be  given  a  set  of  materials  which  has  an  identification  number 
on  each  page.  This  number  will  be  used  eolaly  to  correlate  the  various  sections. 
To  Insure  confidentiality,  please  DO  NOT  write  your  name  on  any  page. 

Your  participation  in  this  study  will  be  limited  to  this  one  seesion. 

There  will  be  no  follow-ups  of  any  kind. 
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DECISION-MAKING  STUDY:  EXAMPLES 


General  Description 

The  following  pages  contain  three  examples  similar  to  those  for  which 
you  will  be  asked  to  make  probability  estimates.  In  the  first  example  you 
are  asked  to  estimate  a  probability  for  a  situation  with  which  you  have 
some  general  familiarity. 

In  the  second  and  third  examples,  an  Initial  set  of  conditions  suggests 
that  the  required  probability  estimate  Is  50X.  Then  additional  Information 
Is  presented.  Some  of  this  Information  may  tend  to  suggest  an  Increase  In 
the  probability  estimate,  while  some  may  suggest  a  decrease  or  no  change  at 
all.  Your  task  is  to  evaluate  all  of  the  Information  and  judge  for  yourself 
what  the  net  effect  would  be. 

Remember,  you  are  to  express  your  probability  estimate  as  a  percentage 
between  OX  and  100X. 
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EXAMPLE  A 


Note: 


There  were  over  three  Billion  births  in  the  United  States  in  1974. 

Question:  What  would  you  estimate  the  probability  to  be  that 
a  baby  born  in  the  United  States  in  1974  was  born 
east  of  the  Mississippi  River? 


Answer : 


Z 


This  problem  has  to  do  with  your  estimate  of  whether  in  1974 
more  babies  were  born  east  of  the  Mississippi  than  west  of  it. 

If  you  think  so,  your  estimate  ahould  be  greater  than  SOZ.  If 
you  think  there  were  leas  born  eaat  of  the  Mississippi  your 
estimate  should  be  lees  than  SOZ.  Of  course,  if  you  believe  an 
equal  number  of  babies  were  born  on  each  side  of  the  Mississippi, 
your  probability  estimate  should  be  SOZ. 
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EXAMPLE  B 


At  the  teller's  window  of  local  banka  there  are  various  types  of  trans¬ 
actions  including  checking  depoaita,  savinga  deposits  and  withdrawals,  U.S. 
Savings  Bonds  purchases,  traveller's  check  purchases,  loan  payments,  check 
cashing,  making  change,  payment  of  utility  bills,  and  various  combinations 
of  these  and  other  transactions.  The  most  common  type  of  transaction  is 
cashing  a  check  drawn  on  one's  own  account.  In  fact,  local  banka  have  found 
that  SOX  of  all  teller  transactions  involve  cashing  such  a  check. 

Thus,  it  1 s  reasonable  to  estimate  that  the  probability  is  SOX  that  a 
teller  transaction  in  a  local  bank  will  Involve  cashing  a  check  on  one's 
own  account. 


However  such  transactions  are  not  equally  distributed  with  respect  to 
age  and  sex  of  the  customer  or  day  of  the  month.  Some  types  of  customers 
are  more  likely  to  cash  their  own  checks  than  others,  and  particularly  so  at 
certain  times  and  at  certain  offices. 

Assume  you  are  in  line  at  a  teller  window  in  one  of  the  bank  offices 
within  one  block  of  campus.  Suppose  it  is  a  Friday  morning  early  in  the 
month,  when  most  checking  accounts  usually  have  a  plus  balance. 

The  man  Just  ahead  of  you  in  line  appears  to  be  of  college  age.  As 
far  as  you  can  tell  he  is  holding  only  one  item,  a  check  bearing  the  name 
of  this  bank.  However,  you  cannot  tell  how  it  is  made  out  or  endorsed. 


Question:  What  would  you  estimate  the  probability  to  be  that  the 

peraon  ahead  of  you  is  cashing  a  check  on  his  own  account? 

Susmary 


Initial  probability  estimate:  SOX 
Inf ormatlon: 

(1)  Bank  near  campus 

(2)  Friday  morning  early  in  month 

(3)  College  age  man 

(4)  Check  on  this  bank 


Answer : 


X 


Note:  If  you  feel  that  the  time,  place  and  appearance  of  the  person  tends 

to  suggest  that  It  is  a  student  cashing  his  own  check,  you  should 
revise  the  probability  estlsuta  upward.  On  the  other  hand,  if  you 
feel  the  information  tends  to  suggest  it  is  not  a  student  cashing  hla 
own  check,  you  should  revise  the  probability  estimate  downward.  The 
degree  to  which  you  revise  the  estimate  will  be  based  on  your  own  feel¬ 
ings  about  the  information. 
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LXAHl'LL  t 


Assune  time  cticrc  arc  two  large  Identical  bags,  "B"  and  "R",  each 
(tiled  with  a  large  number  of  poker  chlpa.  Although  each  bag  contains 
the  sane  number  of  chips,  there  la  one  difference: 

T.ag  ’*S"  contains  7SX  blue  chlpa  and  25X  red  chips 

Dag  "R"  contains  25Z  blue  chips  and  7SX  red  chips 

Based  on  tlie  toss  of  a  coin,  one  of  the  bags  la  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  is  equally  likely  to  hove 

been  selected  by  the  coin  toss.  It  Is  reasonable  to  estimate  that  the 

probability  Is  SOX  that  hag  ”B"  was  chosen. 

Assuoe  that  a  person  who  .is  securely  blindfolded  draws  chips  f  ron 
the  bag.  Before  a  draw  Is  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  is  drawn.  The  chip  is  blue.  The  chip  is 
returned  to  the  selected  bsg,  the  contents  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  is  blue.  The  chip  is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  la  drawn.  Thla  chip 
Is  red. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  baR  is  bag  "B"? 


Sunmiry 


Initial  probability  estimate:  SOX 
Informat  Ion : 

(1)  First  chip  is  blue 

(2)  Second  chip  la  blue 
(})  Third  chip  Is  red 


Answer 


X 


Note: 

Since  more  blue 
to  estimate  that 
than  SOX  because 


chips  were  drawn  from  the  bag.  It  Is  reasonable 
the  probability  that  bag  "B"  waa  choaan  Is  graatar 
that  bag  contains  smtre  blua  chips  than  bag  "ft”  docs. 
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Ident.  No. 


DEC I SION- MAE INC  STUDY:  PROBLEM  SITUATIONS 

The  following  pages  contain  thirty-one  problem  situations  which  are 
similar  to  the  three  examples  Just  presented.  For  each  of  these  situations 
you  will  be  esked  to  estimate  a  probability.  These  estimates  will  serve 
as  date  for  aaalyalt  of  the  variability  In  probability  estimates  made  by 
different  people. 

The  first  fourteen  of  the  problem  situations  are  of  the  "bag-and-chlp" 
variety  which  you  encountered  In  Example  C.  Although  each  bag-and-chlp 
problem  Is  similar  In  form,  you  must  pay  particular  attention  to  (1)  the 
color  composition  of  chips  In  each  of  the  two  bags,  (2)  the  results  of  the 
draws  from  the  chosen  bag,  and  (3)  whether  It  la  bag  "B"  or  bag  "R"  that 
Is  referred  to  In  the  question. 
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Problem  Situation  >1 


As sune  that  there  are  two  large  identical  bags,  "B”  and  "R",  each 

filled  with  a  large  number  of  poker  chlpa.  Although  each  bag  containa 

the  sane  number  of  chips,  there  Is  one  difference: 

Bag  "B"  containa  66*  blue  chips  and  342  red  chlpa 

Bag  "R"  containa  341  blue  chlpa  and  66*  red  chips 

Based  on  the  toas  of  a  coin,  one  of  the  bags  la  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  la  equally  likely  to  have 

been  selected  by  the  coin  toas,  It  la  reasonable  to  estimate  that  the 
probability  la  60S  that  bag  "R"  waa  chosen. 

Assume  that  a  person  who  la  securely  blindfolded  draws  chlpa  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 

thoroughly  nixed.  Then  one  chip  la  drawn.  The  chip  la  red.  The  chip  la 

returned  to  the  selected  bag,  the  contents  are  again  mimed  and  a  second 
chip  la  drawn.  This  second  chip  la  red. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  la  bag  "knT 

Summary 


Initial  probability  eatlmata:  502 
Inf  ormat Ion : 


(t)  First  chip  la  red 
(2)  Second  chip  la  red 
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Problem  Situation  #2 


Assume  that  there  are  two  large  Identical  bags,  "B"  and  "I",  each 
filled  with  a  large  number  of  poker  chlpa.  Although  each  bag  contalna 
the  saae  number  of  chips,  there  la  one  difference: 

Bag  "B"  contalna  75Z  blue  chips  and  23Z  red  chlpa 

Bag  "R"  contains  23Z  blue  chips  and  7SZ  red  chips 

Based  on  the  toaa  of  a  coin,  one  of  the  bags  la  selected.  The  other 

bag  Is  aet  aside.  At  this  point,  since  each  bag  Is  equally  likely  to  have 

been  selected  by  the  coin  toss,  It  is  rsasonsblc  to  estimate  that  the 
probability  Is  50Z  that  bag  "R"  was  chosen. 

Assume  that  a  person  who  Is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  Is  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  Is  drawn.  The  chip  Is  blue.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  srs  aga.  *d  and  a  second 

chip  Is  drawn.  This  second  chip  Is  blue.  The  chip  is  n  med  to 
the  bag,  the  contents  are  remixed  and  a  third  chli  is  draw  This  chip 
Is  blue.  The  chip  Is  returned  to  the  bag,  the  content*  a:  t  mixed 
and  a  fourth  chip  la  drawn.  This  chip  Is  red.  The  lp  Is  returned 
to  the  bag,  the  contents  are  remixed  and  a  fifth  chip  ■  drevn. 

Tills  chip  Is  blue. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  Is  bag  "R"? 


Sunaa ry 

Initial  probability  estimate:  302 
Information: 

(1)  First  chip  la  blue 

(2)  Second  chip  la  blue 
())  Third  chip  Is  blue 
(1)  Fourth  chip  Is  red 

(3)  Fifth  chip  Is  blue 


Answer : 


Z 
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Problem  Situation  #3 


Assume  that  there  are  two  large  Identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  same  number  of  chips,  there  Is  one  difference: 

Bag  "B”  contains  70Z  blue  chips  and  302  red  chips 

Bag  "R"  contains  302  blue  chips  and  702  red  chipe 

Baaed  on  the  toss  of  a  coin,  one  of  the  bags  Is  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  beg  Is  equally  likely  to  have 

been  selected  by  the  coin  toss,  It  Is  reasonsble  to  estimate  that  the 
probability  la  302  that  bag  ”R"  was  chosen. 

Assume  chat  a  person  who  la  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  beg  are 
thoroughly  mixed.  Then  one  chip  Is  drawn.  The  chip  Is  red.  The  chip  le 
returned  to  the  selected  bag,  the  contents  sre  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  blue.  The  chip  la  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  le  drawn.  This  chip 
Is  blue.  The  chip  Is  returned  to  the  bag,  the  contents  are  remixed 
and  a  fourth  chip  la  drawn.  This  chip  la  blue. 

Ouesdon:  what  would  you  eatiaate  to  be  the  probability  that  the 
aelccted  bag  la  bag  ”R”? 


Sum»  ry 

Initial  probability  eatlmata:  302 
Inf ormat Ion: 

(1)  Flrat  chip  Is  red 

(2)  Second  chip  Is  blue 

(3)  Third  chip  Is  blue 

(4)  Fourth  chip  la  blue 


Answer : 


2 
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1'roblea  Situation  #4 


Assure  that  there  are  two  large  Identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chlpa.  Although  each  bag  contains 
the  sare  number  of  chip*,  there  la  one  difference: 

Bag  "B"  contalna  522  blue  chlpa  and  482  red  chlpa 

Dag  'Tv"  contalna  482  blue  chlpa  and  522  red  chlpa 

Baaed  on  the  toaa  of  a  coin,  one  of  the  baga  la  aelected.  The  other 

bag  la  act  aalde.  At  thla  point,  elnce  each  bag  la  equally  likely  to  have 

been  aelected  by  the  coin  toaa.  It  la  reaaonable  to  estimate  that  the 
probability  la  502  that  bag  ”R"  waa  chosen. 

Aasune  that  a  person  who  is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  ons  chip  la  drawn.  The  chip  la  blue.  The  chip  la 
returned  to  the  aelected  bag,  the  contents  are  again  mixed  and  a  second 
chip  la  drawn.  Thla  second  chip  la  blue. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  la  bag  "R"? 


Sunnary 

Initial  probability  estimate:  502 
Inforaat Ion: 

(1)  first  chip  la  blue 

(2)  Second  chip  Is  blue 


Answer : 


2 
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Ident.  No. 


T rob  1 cm  Situation  #5 


As  a  urn;  that  there  are  two  large  Identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chlpa.  Although  each  bag  contalna 
the  aame  number  of  chlpa,  there  la  one  difference: 

Dag  ”B"  contalna  78Z  blue  chlpa  and  22Z  red  chlpa 

Hag  "R"  contalna  22Z  blue  chlpa  and  78Z  red  chlpa 

Baaed  on  the  toaa  of  a  coin,  one  of  the  bege  la  eelected.  The  other 

bag  la  aet  aside.  At  thla  point,  since  each  bag  la  equally  likely  to  have 

been  selected  by  the  coin  toaa,  It  la  reasonable  to  eatloMte  that  the 
probability  la  SOZ  that  bag  "B"  was  chosen. 

Asautec  that  a  person  who  la  securely  blindfolded  draws  chlpa  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  le  drawn.  The  chip  Is  blue.  The  chip  la 
returned  to  the  eelected  bag,  the  contents  ere  again  Mixed  and  a  second 
chip  la  drawn.  Thla  second  chip  la  red.  The  chip  le  returned  to 
the  bag,  the  contents  arc  remixed  and  a  third  chip  le  drawn.  Thla  chip 
la  red. 

Question:  Jhat  would  you  estimate  to  be  the  probability  that  the 
selected  bag  Is  bag  "B"7 


Summary 

Initial  probability  eatlmats:  SOI 
Informat  ton : 

(1)  First  chip  Is  blue 

(2)  Second  chip  Is  red 
(1)  Third  chip  Is  red 


Anawe  r : _ Z 
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Ident.  No. 


Problem  Situation  #6 


Assume  that  there  are  two  large  Identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  sane  number  of  chips,  there  Is  one  difference: 

Bag  "B"  contains  70S  blue  chips  and  30X  red  chips 

Bag  "R"  contains  30X  blue  chips  and  70S  red  chips 

Based  on  the  toss  of  a  coin,  one  of  the  bags  Is  selected.  The  other 

bag  Is  sec  aside.  AC  this  point,  since  each  bag  Is  equally  likely  to  have 

been  selected  by  the  coin  toss.  It  Is  reasonable  to  estimate  that  the 
probability  Is  SOS  that  bag  "B"  was  chosen. 

Assume  chat  a  person  who  is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  drew  is  made,  the  contents  of  tha  selected  bag  are 
thoroughly  mixed.  Then  one  chip  Is  drawn.  The  chip  is  blue.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  blue.  The  chip  Is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  Is  drawn.  This  chip 
Is  red.  The  chip  Is  returned  to  the  bag,  the  contents  ere  remixed 
and  a  fourth  chip  Is  drawn.  This  chip  Is  blue.  The  chip  la  returned 
to  the  bag,  the  contents  are  remixed  and  a  fifth  chip  Is  drawn. 

This  chip  la  blue. 

(Question:  What  would  you  estimate  to  be  the  probability  that  the 

selected  bag  Is  bag  "B"? 

Summary 

Initial  probability  estimate:  50X 
Information: 

(1)  Tlrst  chip  Is  blue 

(2)  Second  chip  Is  blue 

(3)  Third  chip  Is  red 

(4)  Fourth  chip  Is  blue 

(3)  Fifth  chip  Is  blue 


Answer: _ X 
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Idem.  No. 


Problem  SI tuatlon  t 7 


Assume  that  there  are  two  large  Identical  bags,  "B"  and  “it",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  sane  number  of  chips,  there  la  one  difference: 

Bag  ”B"  contains  672  blue  chips  and  332  red  chipa 

Bag  "R"  contains  332  blue  chips  and  672  red  chips 

Based  on  the  toas  of  a  coin,  one  of  the  bags  ia  selected.  The  other 

bag  Is  set  aside.  At  thla  point,  since  each  bag  ia  equally  likely  to  have 

been  selected  by  the  coin  toea,  it  la  reasonable  to  estimate  that  the 
probability  la  302  that  bag  "B"  waa  chosen. 

Assume  that  a  person  who  la  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 
thoroughly  mixed.  Then  one  chip  la  drawn.  The  chip  la  rad.  The  chip  la 
returned  to  the  selected  bag,  the  contents  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  red.  The  chip  is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  la  drawn.  This  chip 
Is  red. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bap.  la  bag  "B"? 


rl 

Initial  probability  estimate:  302 
Information: 

(1)  First  chip  la  red 

(2)  Second  chip  la  red 

(3)  Third  chip  la  red 


Answer : 


2 


Ident.  No. 


Problem  Situation  #8 


Assume  that  there  are  two  large  Identical  baga,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chlpe.  Although  each  bag  contains 
the  same  number  of  chips,  there  la  one  difference: 

Bag  "B"  contains  542  blue  chips  and  462  red  chips 

Bag  "R"  contains  462  blue  chips  and  542  red  chips 

Based  on  the  toss  of  a  coin,  one  of  the  bags  is  selected.  Ttie  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  is  equally  likely  to  have 

been  selected  by  the  coin  toss,  it  Is  reasonable  to  estimate  that  the 
probability  is  502  that  bag  "R"  was  chosen. 

Assure  that  a  person  who  la  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 
thoroughly  mixed.  Then  one  chip  is  drawn.  The  chip  la  red.  The  chip  Is 
returned  to  the  selected  hag,  the  contents  are  again  mixed  and  a  second 
chip  Is  drswn.  This  second  chip  Is  red.  The  chip  Is  returned  to 
the  bag,  the  contents  are  rtnlxsd  and  a  third  chip  is  drawn.  This  chip 
Is  blue. 

Question:  What  would  you  eatlnate  to  be  the  probability  that  the 
selected  beg  is  bag  "PM? 


Summary 

Initial  probability  estimate:  502 
Information; 

(1)  First  chip  la  red 

(2)  Second  chip  is  red 
(J)  Third  chip  la  blue 


Ident.  No. 


Problem  Sltuat Ion  f  9 


Aasusse  that  there  are  two  large  Identical  bags,  "B"  and  "R"t  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  sane  nunber  of  chips,  there  is  one  difference: 

Bag  "B"  contains  69Z  blue  chips  and  lit  red  chips 

Bag  "R"  contains  HZ  blue  chips  and  B9Z  red  chips 

Baaed  on  the  toss  of  a  coin,  one  of  the  bags  is  selected.  The  other 

bag  is  set  aside.  At  this  point,  since  each  bag  is  equally  likely  to  have 

been  selected  by  the  coin  toss.  It  Is  reasonable  to  estimate  that  the 
probability  Is  50Z  that  bag  "B”  was  chosen. 

Assune  that  a  person  who  Is  securely  blindfolded  drews  chips  from 
the  bag.  Before  a  draw  is  made,  the  contents  of  the  selected  bag  are 
thoroughly  mixed.  Then  one  chip  la  drawn.  The  chip  Is  red.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  is  red. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  is  bag  "B"? 


Sufnnary 

Initial  probability  estimate:  50Z 
Information: 

(1)  First  chip  Is  red 

(2)  Second  chip  Is  red 

Answer : _  Z 


" 


Ident.  No. 


Problem  Sltuat Ion  #10 


Assume  that  there  sre  two  large  Identical  bags,  "B"  and  "R",  each 
tilled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  same  number  of  chips,  there  is  one  difference: 

bag  "B"  contains  83X  blue  chips  and  13X  red  chips 

Bag  contains  132  blue  chips  and  832  red  chips 

Based  on  the  toss  of  a  coin,  one  of  the  bags  is  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  la  equally  likely  to  have 

been  selected  by  the  coin  toss,  it  la  reaaonable  to  estimate  that  the 
probability  Is  30Z  that  bag  "8"  was  chosen. 

Assume  that  a  person  who  Is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  is  made,  the  contents  of  the  selected  bag  are 
thoroughly  mixed.  Then  one  chip  is  drawn.  The  chip  is  red.  The  chip  is 
returned  to  the  selected  bag,  the  contcnta  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  blue.  The  chip  is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  is  drawn.  This  chip 
Is  red.  The  chip  Is  returned  to  the  bag,  the  contents  are  remixed 
and  a  fourth  chip  la  drawn.  This  chip  is  blue.  The  chip  la  returned 
to  the  bag,  the  contenta  are  remixed  and  a  fifth  chip  is  drawn. 

This  chip  Is  blue. 

(Juration:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  is  bag  "B"? 


Surmary 

Initial  probability  estimate:  30Z 
Information: 

(1)  First  chip  la  red 

(2)  Second  chip  la  blue 

(3)  Third  chip  la  red 
(A)  Fourth  chip  la  blue 
(3)  Fifth  chip  la  blue 

Answer: _ Z 
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Ident.  No. 


1'roblem  Situation  ^11 


Assume  that  there  are  two  large  Identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  same  nunber  of  chips,  there  is  one  difference: 

Lag  "B"  contains  90Z  blue  chips  and  10Z  red  chips 

Bag  "R"  contains  1 OX  blue  chips  and  90Z  red  chips 

Based  on  the  toss  of  a  coin,  one  of  the  bags  Is  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  Is  equally  likely  to  have 

been  selected  by  the  coin  toss,  It  Is  reasonable  to  estimate  that  the 
probability  Is  50Z  that  bag  "R"  was  chosen. 

Assume  that  a  person  who  is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  Is  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  Is  drawn.  The  chip  la  red.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  are  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  red.  The  chip  la  returned  to 
the  bag,  the  contents  arc  remixed  and  a  third  chip  la  drawn.  This  chip 
Is  blue.  The  chip  Is  returned  to  the  bag,  the  contents  are  remixed 
and  a  fourth  chip  Is  drawn.  This  chip  Is  red. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  Is  bag  ”P,”? 


bunmary 

Initial  probability  estimate:  bOZ 
Inforn.it  Ion: 

(1)  First  chip  Is  red 

(2)  Second  chip  Is  red 
(})  Third  chip  Is  blue 
(4)  Fourth  chip  Is  red 


Answer : 


Z 


Ident.  No. 


Problem  Situation  #12 


Assume  that  there  are  two  large  identical  bags,  "B"  and  "R",  each 
filled  with  a  large  number  of  poker  chlpa.  Although  each  bag  contains 
the  same  number  of  chips,  there  la  one  difference: 

Bag  "B"  contalna  59Z  blue  chlpa  and  A 1 X  red  chlpa 

Bag  ”R"  contalna  4IZ  blue  chlpa  and  59Z  red  chlpa 

Based  on  the  toss  of  a  coin,  one  of  the  baga  Is  selected.  The  other 
bag  Is  set  aside.  At  this  point,  since  each  bag  la  equally  likely  to  have 
been  selected  by  the  coin  toss,  It  is  reasonable  to  estimate  that  the 
probability  la  SOZ  that  bag  "B"  was  chosen. 

Assume  that  a  peraon  who  la  securely  blindfolded  drawa  chips  from 
the  bag.  Before  a  draw  la  made,  the  contents  of  the  selected  bag  are 
thoroughly  mixed.  Then  one  chip  la  drrvn.  The  chip  la  blue.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  are  again  nixed  and  a  second 
chip  Is  drawn.  This  second  chip  la  red.  The  chip  la  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  la  drawn.  This  chip 
Is  red.  The  chip  la  returned  to  the  bag,  the  contents  are  remixed 

and  a  fourth  chip  la  drawn.  This  chip  la  red. 

Question:  What  would  you  eatlmate  to  be  the  probability  that  the 
selected  bag  Is  bag  "B"? 


Suarvary 


Initial  probability  eatlmate:  50Z 
Inf  omit  Ion : 

(1)  First  chip  la  blue 

(2)  Second  chip  la  red 

(3)  Third  chip  is  red 

(4)  Fourth  chip  la  red 


Answer : 


Z 
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Ident .  No. 


1'roblcn  Situation  #13 


Assume  that  there  are  two  large  identical  bags,  "B"  and  ”R",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  same  nunber  of  chips,  there  Is  one  difference: 

Bag  ”1!”  contains  65Z  blue  chips  and  35X  red  chips 

Bag  *H"  contains  35X  blue  chips  and  65Z  red  chips 

Based  on  die  toss  of  a  coin,  one  of  the  bags  Is  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  is  equally  likely  to  have 

been  selected  by  the  coin  toss.  It  Is  reasonable  to  estimate  that  the 
probability  Is  30Z  that  bag  ”R"  was  chosen. 

Assume  that  a  person  who  Is  securely  blindfolded  draws  chips  from 
the  bag.  Before  a  draw  Is  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  Is  drawn.  The  chip  Is  red.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  are  again  mixed  and  a  second 
chip  la  drawn.  This  second  chip  la  red.  The  chip  Is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  Is  drawn.  This  chip 
Is  red. 

tjuestlon:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  Is  bag  ”R"T 


Summary 

Initial  probability  estimate:  30Z 
Inf ormat Ion : 

(1)  First  chip  Is  red 

(2)  Second  chip  Is  red 

(3)  Third  chip  Is  red 


Answer : _ Z 
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Problem  Situallon  #14 


Ident.  No. 


Assure  that  there  arc  two  large  Identical  bags,  "B"  and  "P",  each 
filled  with  a  large  number  of  poker  chips.  Although  each  bag  contains 
the  sane  number  of  chips,  there  Is  one  difference: 

Bag  "B"  contains  67Z  blue  chips  and  33Z  red  chips 

Bag  "H"  contains  33Z  blue  chips  and  67Z  red  chips 

Based  on  the  toss  of  a  coin,  one  of  the  bags  Is  selected.  The  other 

bag  Is  set  aside.  At  this  point,  since  each  bag  Is  equally  likely  to  have 

been  selected  by  the  coin  toss,  It  Is  reasonable  to  estimate  that  the 
probability  Is  501,  that  bag  "P."  was  chosen. 

Assune  that  a  person  who  Is  securely  blindfolded  draws  chips  [ron 
the  bag.  Before  a  draw  Is  made,  the  contents  of  the  selected  bag  are 
thoroughly  nixed.  Then  one  chip  Is  drawn.  The  chip  Is  red.  The  chip  Is 
returned  to  the  selected  bag,  the  contents  arc  again  mixed  and  a  second 
chip  Is  drawn.  This  second  chip  Is  red.  The  chip  Is  returned  to 
the  bag,  the  contents  are  remixed  and  a  third  chip  Is  drawn.  This  chip 
Is  blue. 

Question:  What  would  you  estimate  to  be  the  probability  that  the 
selected  bag  Is  bo;;  "R"? 


Sumnarv 

Initial  probability  estimate:  501 

Informat  Ion: 

(1)  Plrst  chip  Is  red 
(7)  Second  chip  Is  red 
(3)  Third  chip  Is  blue 


Answer : 
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Ident.  No. 


Problem  Situation 


Assume  that  GM  sells  the  same  number  of  1978  trucks  that  are  sold 
by  non-GM  manufacturers.  Thus,  If  in  a  conversation  with  a  stranger, 
you  found  out  that  he  had  purchased  a  new  1978  truck,  it  would  be  reason¬ 
able  to  estimate  that  the  probability  is  50X  that  his  truck  was  a  GM. 

Suppose  that,  at  the  beginning  of  the  nest  model  year,  GM  sustains 
a  massive  strike  for  three  months  during  which  no  1979  trucks  are  produced, 
while  all  non-GM  truck  manufacturers  avoid  strikes.  Further  suppose  that, 
midway  through  the  model  year,  a  Ralph  Nader-type  group  issues  a  report 
that  1979  CM  trucks  have  serious  safety  defects.  In  addition,  assume 
that  although  1979  GM  trucks  Increased  in  price  by  an  average  of  $190  over 
the  previous  year's  price,  non-GM  trucks  Increased  in  price  by  an  average 
of  $345. 


Soswtlme  in  1980  you  meet  another  stranger  whom  you 
a  new  1979  truck. 


find  has  purchased 


Question:  What  would  you  eatlmste  to  be  the  probability  that  it 
was  a  GM  truck? 


Initial  probability  estimate:  SOX 


Informat  ion : 


(1)  GM  strike 

(2)  GM  safety  questioned 

(3)  Smaller  GM  price  Increase 


I dent.  No. 


Problem  Situation  #16 


You  are  considering  taking  a  particular  elective  course  next  term. 
When  you  first  asked  about  it  a  few  weeks  ago,  the  head  of  the  department 
offering  the  course  told  you  that  it  might  or  might  not  be  given,  and 
that  there  was  s  50Z  probability  that  it  would  be  given  next  term. 

A  few  weeks  later  you  pass  the  department  head  In  the  hall.  He  tells 
you  that  the  Instructor  who  taught  the  course  last  year  will  be  on  leave 
next  term.  You  also  see  a  notice  on  the  bulletin  board  requesting  persons 
Interested  in  this  particular  course  to  sign  up.  The  notice  is  about  a 
week  old  and  there  are  only  three  signatures  on  the  notice.  In  addition, 
you  read  In  the  paper  that  because  of  budget  problems,  some  elective 
courses  may  not  be  offered  next  term,  but  postponed  until  later. 

Question:  In  light  of  these  factors,  what  would  you  estimate  to 
be  the  probability  that  the  elective  course  you  are 
Interested  in  will  be  given  next  term! 

Susaary 

Initial  probability  estimate:  SOZ 
Informal  ion : 

(1)  Instructor  on  leave 

(2)  Few  signatures  on  sign-up  sheet 

(3)  Postponement  policy  announced 


Answer: _ X 
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Jdent.  No. 


Problem  Situation  #17 


There  la  a  atreet  corner  in  town  where  a  "local"  etreet  lnteraects 
a  "through"  atreet  In  a  "T".  The  local  atreat  doaa  not  continue  at 
thla  point,  ao  only  turna  onto  the  through  atraet  are  poaalble  for 
cara  arriving  at  thla  lnteraectlon  from  the  local  atreet.  Left  turna 
lead  to  the  unlveralty  caapua,  while  right  turna  lead  to  ahopplng 
centara.  During  an  average  weekday,  aa  aany  cara  arriving  at  thla 
lnteraectlon  froa  the  local  atreat  turn  left  aa  turn  right.  Thua,  It 
la  reaeonable  to  eetlaate  that  the  probability  la  50X  that  a  car 
arriving  at  thla  lnteraectlon  will  turn  left. 


THROUGH  STREET 


*  To  caapua 


Suppoae  you  approach  that  lnteraectlon  froa  the  local  atreet  at 
10:45  a. a.  on  a  weekday  whan  unlveralty  claaaea  are  In  aaaalon.  You 
note  that  the  car  ahead  of  you  hae  a  faculty/ataf f  parking  atlcker  on 
the  rear  buaper.  A  uoaan  driver  la  the  aole  occupant. 

Queatlon:  What  would  you  eatlaate  to  be  the  probability  that 
thl*  particular  car  will  turn  left? 


Initial  probability  eatlaate:  50Z 
Information: 

(1)  10:45  a. a. 

(2)  unlveralty  claaaea  in  aaaalon 
())  faculty/ataf f  atlcker 

(4)  woman  driver 


IJent.  No. 


Problem  Situation  #18 


Although  Springfield  is  the  capital  of  Illlnola,  the  population  of 
that  state  is  concentrated  around  Chicago,  which  Is  located  approximately 
200  miles  northeast  of  Springfield.  In  fact,  the  population  of  the  Chicago 
metropolitan  area  la  almost  exactly  half  the  total  population  of  the  state 
of  Illinois.  Thus,  It  la  reasonable  to  estimate  that  the  probability  la 
50X  that  a  person  from  Illinois  Is  from  the  Chicago  metropolitan  area. 

Suppose  that  on  a  nonstop  plane  from  Washington,  D.  C.  to  Chicago  you 
meet  a  stranger  who  says  he  lives  In  Illinois.  He  mentions  that  he  worka 
for  the  State  of  Illlnola.  He  also  mentions  that  he  la  a  sports  fan  and 
attends  many  of  the  Chicago  baseball  games.  When  you  arrive  In  Chicago 
and  retrieve  your  carry-on  luggage  you  note  that  his  bag  has  several 
Springfield,  Ill.,  atlckare.  He  Is  alao  carrying  parcels  that  are  labelled 
"Illinois  Department  of  Agriculture." 

Question:  What  would  you  estimate  to  be  the  probability  chat  this 
stranger  resides  In  an  area  of  Illinois  other  than  the 
Chicago  metropolitan  area? 

b£*SIl 

Initial  probability  estimate:  SOX 
Information : 

(1)  Works  for  State  of  Illlnola 

(2)  Attends  Chicago  sports  events 

(3)  Bag  has  Springfield  stickers 

(4)  Parcels  say  "Illinois  Department  of  Agriculture" 


Answer : 


l 


-83- 


Ident.  No. 


Problem  Situation  019 


In  New  York  harbor,  Boat  of  the  passenger  facilities  are  in  Manhattan. 
Brooklyn,  on  the  other  hand,  has  the  majority  of  the  large  cranes  which 
facilitate  unloading  heavy  deck  cargo.  Normally  about  half  the  ships  which 
enter  New  York  harbor  from  the  Atlantic  dock  in  Manhattan.  Moat  of  the 
remainder  dock  in  New  Jersey  or  in  Brooklyn.  Thus  it  is  reasonable  to  esti¬ 
mate  that  the  probability  is  50X  that  a  ship  entering  New  York  harbor  from 
the  Atlantic  will  dock  in  Manhattan. 

Suppose  you  have  read  in  the  paper  that  ships  docking  in  New  Jersey 
are  currently  experiencing  extended  delays  in  unloading  due  to  a  workers' 
slowdown.  Today,  as  you  cross  the  bridge  at  the  New  York  harbor  entrance, 
you  see  a  large  cargo  ship  entering.  This  ship  appears  to  be  loaded  with 
heavy  deck  cargo. 

Question  What  would  you  estimate  to  be  the  probability  that  this 
ship  will  dock  in  Manhattan? 


Suaamry 

Initial  robability  estimate:  SOX 


Inf c  rmat Ion: 

(1) 

New  Jersey 

slowdown 

(2) 

Cargo  ship 

(3) 

Heavy  deck 

cargo 

Answer: _ _ _ X 
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Ident.  No. 


Problem  Situation  #20 


On  an  average  day  tba  local  AAP  supermarkets  aell  the  same 

amount  of  Pepai  and  Coke - tome  In  cans,  some  In  throwaways,  and 

some  In  returnable  bottlea.  Thus,  It  la  reasonable  to  eetlmete 
that  the  probability  ia  50Z  that  a  local  AAP  customer  who  buys 
one  of  the  two  leading  colas  vlll  buy  Pepsi. 

This  week  the  local  AAP  is  having  a  sale  in  which  8-packs  of 
Pepsi  returnable  16-ounce  bottlea  are  on  sale  for  50c  less  per 
carton  than  Coke.  Furthermore,  the  store  has  a  special  display  of 
Pepsi  products  near  the  entrance.  However,  a  nearby  supermarket 
competitor  is  featuring  a  50c  per  carton  reduction  on  8-packs  of 
Coke,  announced  in  a  large  window  sign  and  prominently  featured  in 
newspaper  ads. 

Question:  In  light  of  this  information,  what  would  you  esti¬ 
mate  to  be  the  probability  that  a  local  AAP  shopper 
who  buys  one  of  the  leading  colas  this  week  buys 
Pepsi  Instead  of  Coke? 


Initial  probability  estimate:  50X 
Information: 

(1)  AAP  Pepsi  sale 

(2)  AAP  Pepsi  display 

(3)  Competitor  Coke  sals 


Answer :  _ X 
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Icient.  No. 


Problem  Situation  #21 


There  were  alnoat  80  million  votes  cast  In  the  U.S.  presidential 
election  In  November  1976.  Jimmy  Carter  received  approximately  40 
all lion  votes.  Thus.  It  Is  reasonable  to  estimate  that  the  prob¬ 
ability  Is  SOX  that  a  person  who  voted  in  the  1976  presidential 
election  voted  for  Carter.  However,  Carter  received  56  more 
electoral  votes  than  Ford  and  was  elected.  Carter  ran  stronger  in 
some  regions  than  others  and  was  more  popular  with  some  elements 
of  the  electorate  than  others. 

Suppose  that  on  a  trip  to  Ceorgla  you  meet  a  black  woman  who 
tells  you  she  Is  53  years  old  and  has  lived  In  Ceorgla  all  her  life. 
You  also  learn  from  her  that  she  la  married  and  works  In  a  textile 
alll.  She  mentions  that  she  voted  In  the  1976  presidential 
election. 

Question:  What  would  you  estimate  to  be  the  probability  that 
this  woman  voted  for  Jimmy  Carter? 


Initial  probability  estimate:  50X 
Information : 


(1) 

Black  woman 

(2) 

53  years  old 

(3) 

Lifetime  Ceorgla 

reeldent 

(4) 

Married 

(6) 

Works  In  textile 

mill 

Answer:  _ X 
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Idem.  No. 


Problem  Situation  #22 


Roughly  half  the  bablea  born  In  thla  country  are  glrla,  ao  lacking 
other  Information  it  la  raaaonable  to  eatlmate  that  the  probability  Is  SOX 
that  a  particular  birth  In  the  local  hospital  will  be  a  girl. 

Mrs.  Jones  Is  nearing  the  claw  for  her  baby  to  be  born.  Her  doctor  thinks 
It  may  be  a  girl  because  Its  heartbeat  la  somewhat  faster  than  the  average 
for  boys.  The  nurse,  noting  that  the  Jonea  have  three  boys,  thinks  It  will 
be  a  boy  because  she  feels  that  boys  run  In  their  family.  Mr.  Jones,  a 
casual  dice  player,  feels  It  will  be  a  girl  because  "it  Is  time  for  the  run 
to  change,"  thus  helping  to  even  the  trend. 

Question:  What  would  you  estimate  to  be  the  probability  that  Mrs. 

Jones'  baby  will  be  a  girl? 


Initial  probability  estimate:  50X 
Information: 

(1)  Paster  heartbeat 

(2)  Nurse's  opinion  that  boys  run  In  family 

(3)  Father's  opinion  that  a  girl  would  help  even  trend 


Answer : 


Z 
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Problem  Situation  #23 


Assume  you  are  in  class  on  the  fourth  floor  of  •  seven  story  class¬ 
room  building,  so  there  are  three  floors  above  this  floor  and  three  floors 
below.  You  leave  your  class  a  half  hour  early  because  of  a  dental  appoint¬ 
ment  and  go  to  the  only  public  elevator.  The  elevator  Is  not  there,  so 
you  push  the  button.  On  the  average,  in  this  type  of  situation,  the 
elevator  is  as  likely  to  have  been  left  by  the  previous  user  on  a  floor 
above  as  on  a  floor  below.  Thus,  it  la  reasonable  to  estimate  that  the 
probability  is  50X  that  the  elevator  will  come  up  from  below  in  response 
to  your  call. 

However,  in  this  case  you  realise  that  you  are  not  taking  into  account 
all  of  the  circumstances.  in  this  building  the  seventh  floor  is  primarily 
instructors'  offices,  the  sixth  floor  is  mostly  research  laboratories, 
and  the  remaining  floors  are  all  classrooms.  Furthermore,  half  of  the 
third  floor  classrooms  are  being  painted.  In  addition,  it  is  early  in  the 
morning  since  it  was  your  first  period  class  that  you  left. 

Question:  Under  these  circumstances  what  would  you  estimate  to  be 
the  probability  that  the  elevator  will  come  up  from 
below  in  response  to  your  call? 


Initial  probability  estimate:  50Z 


Information : 


Seventh  floor  offices 

Sixth  floor  labs 

Other  floors  are  classrooms 

Half  of  third  floor  rooms  being  painted 

Horning,  middle  of  first  period 


Answer : 
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Ident.  No. 


Problaa  Situation  #24 


New  York  scat*  la  the  largest  patron  of  the  arta,  leading  all  other 
atatea  in  1975  In  tenaa  of  atata  appropriatlona  for  the  arta. 

Question:  What  would  you  estimate  to  be  the  probability  that  a  dollar 
appropriated  for  the  arte  by  one  of  the  atatea  in  1975  came 
from  a  New  York  atate  appropriation? 


Answer: 


: 


idem..  No. 


Problem  Situation  #25 


Philadelphia  la  cha  largest  city  In  Pennsylvania  In  population, 
according  to  the  1970  census. 

Question:  What  would  you  estimate  to  be  the  probability  that  a 
Pennsylvania  resident  In  1970  was  from  Philadelphia? 


Answer : 


X 
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Ident.  No. 


Thar*  war# 
Qua* t lorn 

Answer : 


Problaa  Situation  *26 


ov*r  16,000  death#  In  th*  U.  S.  In  1976  dua  to  falls. 

What  would  you  satinets  to  b*  th*  probability  that  a  parson 
who  died  In  a  fall  In  th#  0.  S.  in  1974  was  over  54  years 
old? 


X 
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ident.  No. 


Problaa  Situation  #27 


According  to  Federal  figures  there  war#  alaoat  133  Billion  aotor 
vahlclas  registered  la  the  U.  S.  la  1373. 

Question:  What  would  you  aatlaata  to  ba  tha  probability  that  a  motor 

vehicle  la  tha  U.  S.  waa  raglatarad  la  Pennsylvania  in  1975? 


Ident.  No. 


Problem  Situation  #28 


FBI  figures  for  1973  report  that  there  were  116,000  arrests  In  the 
U.S.  that  year  for  robbery. 

Question:  What  would  you  estimate  to  be  the  probability  that  a  1973 
0.  S.  errsst  for  robbery  was  a  sen  rather  than  a  woman? 


Answer : 


X 


Ident .  No. 


Probla*  Situation.  #29 


Automobiles  are  the  single  »<>•*  c°—°°  c*u*«  ot  «cld«t»1  d“th  ln 

u.  s. 


Question : 


Whet  would  you  estimate  to  b*  the  probebility  that  * 
person  who  died  In  en  accident  in  the  U.S.  ln  1974  wee 
killed  ln  en  auto  accident? 


Ident.  No. 


Problem  Situation  #30 


There  were  over  6.7  Billion  new  peeeenger  cere  produced  In  the  U.  S. 
in  1975. 

Queetlon:  Whet  would  you  estimete  to  be  the  probebility  thet  e  1975 
U.  S. -menuf ectured  peeeenger  cer  wee  e  product  of  Ccnerel 
Ho tore 7 


Anewer : 
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Ident.  No. 


Problem  Situation  #31 


There  ere  a  number  of  departments  In  the  U.  S.  federal  government. 

Queation:  What  would  you  estimate  to  be  the  probability  that  a  dollar 
of  Federal  tax  money  budgeted  for  expenditure  in  flacal  year 
1976  was  allocated  for  expenditure  by  the  Department  of  Health 
Education  and  Welfare  rather  than  aome  other  department  of 
the  federal  governsMnt? 


Answer: 


THIS  IS  THE  LAST  PROBLEM  SITUATION.  AT  THIS  POINT,  PLEASE  CHECK 
BACK  TO  SEE  THAT  YOU  HAVE  PROVIDED  ANSWERS  FOR  ALL  31  PROBLEM  SITUATIONS. 


ATTER  YOU  HAVE  DONE  THIS,  PLEASE  COMPLETE  THE  SHORT  QUESTIONS  c  ON 
THE  FOLLOWING  TWO  PACES. 


I 


Idont  Mo  ______ 

tyyruwagyT  .t»wgv>n« 


l.  ion  of  tha  ptoblaaa  IN  Ium  Jaot  MUtlliW  an  lafllal  WJ 

probability  aatlaata  «M(1  too  wto  oakoO  to  tooloo  oa  tho  hoato  of 
o«41ttaaal  loforoailoa.  Tlsaao  loOlcato  bo  loo  baa  oalU  yoa  tklak  tha 
loltlal  MM  probability  aatlaata  aaa  la  oacb  caaa.  Thao  lablcata  bar 
aura  yoa  faal  oboat  tha  probability  aatlaataa  yoa  aa4a  fiaally. 
loOlcoto  boa  raallatlc  yoa  foal  oath  allaatlaa  oaa  flat*  thpcb  mjM 

it  tMnnt  mi  mlmUt* 


rmui 

jiTuarioa 


ba|  aab  Chip  Troklaaa 


Ol  Tracks 


t loci  loo  C oar oo 


ft  root  lototoorttaa 


lllloolt  looiaoat 


too  Tort  borbor 


Topol  oo.  Cabo 


TroolOaoilal  11 or  t i ao 


aro.  Joaot-  baby 


(loootor 


Trobloaa  with  Bo  lolt la 
Trobobllltp  latlaato 


boa  Woo  tail lal  SOI 

ftababllUT  kil— Ul 


I 


si 


Is 


Haw  lata  Ursa  Toa 

of  Toot  Utlaatoat 


5  3 

*3 


a* 


N oa  laalfatlc 
Mpra  lltaationf’ 


1] 


r  x 

S  t 


A 

«a  «• 
m  bo 

•j 


H 


(Reduced  fro*  full-page  else.) 
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Do  you  think  there  night  b«  i  wy  to  calculate  the  bag  and  chip 
probabllltlaa? 

If  so,  how? 


Hava  you  avar,  in  school  or  elsewhere,  an co untar ad  situations  requiring 
you  to: 

a.  Maks  probability  estimates? _ _ 

b.  Revise  probability  astlaatas  on  basis  of  subsequent  infor- 
aatloaf 

How  many  years  of  education  have  you  completed  beyoud  high  school 
(including  the  current  year)? 

Major  field  _ 

Minor  fields _ 

Approximate  overall  grade  point  average _ 

Indicate  the  number  of  credits  (if  aay)  completed  or  currently  scheduled 
la: 

Probability _ Statistics _ Decision  Theory _ 

Age _ 

Sax _ 

Governs: 
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